Skip to content

ODD

An Operational Design Domain (ODD) states the conditions a system is built to drive in: urban roads, up to 60 km/h, no heavy rain, and so on. Every run is measured against one. The runner samples the ODD's attributes on every tick. Their buckets are the run's ODD coverage: which conditions it reached (see Coverage). The ODD's modules say which ticks were outside the ODD.

An ODD is written in one of two ways. Both build the same objects, so they are measured, judged and reported the same way:

  • Python: openodd's model (Writing an ODD in Python), with this framework's probes. It is checked with Codon before it is used, like a scenario (Static Check).
  • ASAM OpenODD 1.0 YAML (the standard's chapter 10). It is read into the same objects. A binding file next to it names the probes that measure its concepts.
 OpenODD YAML + binding file ──(load_odd_binding)──┐
                                                    ├──▶ OddDefinition ──▶ coverage + inside/outside per tick
 Python (checked by Codon) ─────────────────────────┘

OpenODD also maps its model to OpenSCENARIO DSL (chapter 9, model to DSL only) and to tables (chapter 8). Neither is read here.

The reading, judging, versioning, sampling and coverage are openodd (openodd-py, a Rust core with a Python API), which knows nothing of CARLA. This framework is its backend: autoware_carla_scenario.odd.backend.CarlaBackend, a subclass of openodd.Backend, gives the built-in probes a binding file names, measure: keys read from the running scenario, the knobs that set the weather and the sun, and where the ego is (for coverage). Importing autoware_carla_scenario.odd makes it openodd's default backend, so openodd.load_odd_binding() and autoware_carla_scenario.odd.load_odd_binding() read the same.

Versions of an ODD

OpenODD 1.0 defines no version of an ODD; openodd builds one on what it does define (its versioning profile):

  • the revision is the SHA-256 of the files the ODD was read from -- the binding file, the OpenODD files it names and every file they IMPORT -- by name and content. The same files are the same revision on any machine and from any repository or commit; any change to any of them is a new one;
  • the version is what the root modules (those under ODD:) declare in their METADATA (version: 2.1.0);
  • the structure is the digest of the ODD as built (modules, buckets, the buckets outside it), for an ODD written in Python, which has no files.

All three are written to every coverage file (the ODD's revision, version, structure), so coverage can be told apart by the ODD it was measured against. OddDefinition.archive() packs the files into a deterministic .tar.gz that openodd.load_odd_archive() reads back exactly; a run writes it beside its coverage file ({ScenarioName}_odd.tar.gz).

Concepts

The model and its semantics follow OpenODD 1.0 (chapters 6 and 7).

Concept What it is
Attribute A taxonomy concept, measured on every tick: a probe reads it from the world, and its buckets make it a cover item.
Condition A test on one attribute (speed_limit.between(0, 60)), a group of conditions (all_of, any_of), another module's verdict (module_holds), or a missing value (is_unknown).
Module A named rule: INCLUDE AND (NOT EXCLUDE). It has at most one include section and at most one exclude section.
Label A name several modules declare. It holds when any active module declaring it holds.
Root An entry point. Each root candidate (roots=; by default every module) that no other module refers to, by name or through a label, is a root. The ODD holds when its active roots hold.

Missing values. OpenODD's missing-value semantics apply. A value the probe could not read does not, by itself, put a situation outside the ODD (open world). Examples are a simulator without weather, and a run without a Lanelet2 map. When a value is required, the ODD says so with attr.is_unknown() in an exclude section (x: unknown in YAML). A tick that is inside only because values were missing is reported as such.

Inactive modules. An inactive module is ignored. A condition referring to it, either way, drops out of its section, as if it were not written. A section with nothing left in it counts as absent: an include section then holds, and an exclude section does not. The standard says such conditions are "automatically satisfied"; taken literally, that would make an exclude section that refers to a switched-off hazard exclude everything, so this reads "ignored" as the neutral element.

Modules follow ISO 34503's "default" definition mode, as OpenODD requires. Whatever no module rules out is inside the ODD.

Writing an ODD in Python

An ODD is a function that takes nothing and returns an OddDefinition. The model, the registry and the reading of a name are openodd's (its Writing an ODD in Python); this framework re-exports them, gives them its probes, and checks the ODD with Codon:

from __future__ import annotations

import typesafe_carla.carla as carla

from autoware_carla_scenario import (
    OddAttribute,
    OddDefinition,
    OddModule,
    module_holds,
    register_odd,
)
from autoware_carla_scenario.odd import (
    INTENSITY_LEVELS,
    lanelet_location,
    rain,
    speed_limit_kph,
)


def ego_yaw_rate(world: carla.World) -> float | None:
    ...  # any function of the world is a probe; None means unknown


def urban_odd() -> OddDefinition:
    location = OddAttribute(
        "scenery.location", lanelet_location, values=["urban", "nonurban", "private"]
    )
    speed_limit = OddAttribute(
        "scenery.speed_limit", speed_limit_kph, unit="km/h",
        buckets=[0, 30, 40, 50, 60, 80, 100],
    )
    weather = OddAttribute("environment.rain", rain, values=INTENSITY_LEVELS)
    yaw_rate = OddAttribute(
        "dynamic.yaw_rate", ego_yaw_rate, unit="deg/s", range=(-30.0, 30.0), every=10.0
    )
    return OddDefinition(
        "urban",
        [location, speed_limit, weather, yaw_rate],
        [
            OddModule(
                "urban_roads",
                include_and=[location.is_in(["urban"]), speed_limit.between(0, 60)],
                text="Urban roads up to 60 km/h",
            ),
            OddModule("weather", exclude_or=[weather.is_in(["heavy"])], labels=["fair"]),
            OddModule(
                "root", include_and=[module_holds("urban_roads"), module_holds("fair")]
            ),
        ],
        roots=["root"],
        text="Urban roads, fair weather",
    )


register_odd("urban", urban_odd)

Attributes

OddAttribute(name, probe, *, unit, range, every, buckets, values, text) takes the bucket arguments of register_cover(). Give at most one of values, buckets and range (with every). An attribute with none of them is monitored but not covered: conditions may test it, and the report lists it so the blind spot is visible. Its cover item is named "odd." + name.

Conditions

Method Holds when
attr.is_in(values) the value is one of values (compared by label: an enum by name, a bool as true/false)
attr.equals(value) the value equals value
attr.between(low, high) low <= value <= high
attr.at_least(x) / attr.at_most(x) value >= x / value <= x
attr.greater_than(x) / attr.less_than(x) value > x / value < x
attr.is_unknown() the value is missing
all_of([...]) / any_of([...]) all of / any of the conditions hold
module_holds(name, holds=True) the module (or label) named name holds, or does not

Numbers are in the attribute's unit.

Modules

OddModule(name, *, include_and, include_or, exclude_and, exclude_or, labels, active, text, situation, target, cover_by, min_stay). Give at most one of include_and / include_or, and at most one of exclude_and / exclude_or.

Situations

Attribute coverage counts each attribute on its own: three lanes and 60 km/h can both be covered by runs that never drove 60 km/h on a three-lane road. A module already says such a combination, so a module can be covered as a situation:

OddModule(
    "three_lane_cruise",
    include_and=[lanes.equals(3), speed.between(55, 65)],
    situation=True,
    cover_by="meters", target=200, min_stay=3.0,   # 200 m, counting stays of 3 s or more
)

Every tick, the module's verdict (which the ODD works out anyway) is a sample of the item odd.situation.three_lane_cruise, which has one bucket, holds. A tick where the module holds is a sample in it. A tick where it fails, or is inactive, ends the stay. A tick where it rests on missing values ends the stay too, and is counted apart as unknown. target, cover_by and min_stay work as for attributes (docs/coverage.md). The report lists situations in a group of their own, Situation coverage.

A situation is what the runs should drive, not a bound of the ODD: with roots left out, a situation nothing refers to is not a root candidate. To make it a bound too, refer to it from a root, or name it in roots.

Built-in probes

These live in autoware_carla_scenario.odd:

Probe Returns Read from
ego_speed_kph float, km/h Ego velocity
speed_limit_kph float, km/h Lanelet2 speed_limit tag, else CARLA
lanelet_speed_limit_kph float, km/h Lanelet2 speed_limit tag
lanelet_location str Lanelet2 location tag (urban, nonurban, ...)
lanelet_subtype str Lanelet2 subtype tag (road, highway, ...)
road_type highway, or nothing The taxonomy's road type, from the Lanelet2 subtype: highway for a highway; Lanelet2 has no tag that tells a local road from a primary one, so those read nothing
drivable_area_type DRIVABLE_AREA_TYPES The taxonomy's drivable area type, from the Lanelet2 subtype: road and highway a lane, road_shoulder a shoulder, parking a parking_space, play_street mixed_pedestrian_and_vehicular_traffic
in_junction bool Lanelet2 turn_direction tag of the ego's lanelet, else (no Lanelet2 map) the CARLA waypoint
junction_type intersection, or nothing The taxonomy's junction type (the default ODD's): intersection where in_junction reads true; Lanelet2 tells no branch or merge, and outside a junction it reads nothing
lane_count int Driving lanes in the ego's direction, outside junctions: the ego's lanelet and its neighbours in the Lanelet2 routing graph, else (no Lanelet2 map) the CARLA waypoint's OpenDRIVE road
rainfall_rate_mm_h float, mm/h The rainfall rate the run was set to (environment.rainfall_rate_mm_h); nothing when it named none
fog_visibility_range_m float, m The visibility through the fog the run was set to (environment.fog_visibility_range_m); nothing when it named none
time_of_day TIMES_OF_DAY The sun: day above the horizon, dawn (rising, azimuth 0-180) or dusk (setting, 180-360) down to 6 degrees below it, night under that
wind_speed_kmh float, km/h The wind speed the run was set to (environment.wind_speed_kmh); nothing when it named none
cloud_cover CLOUD_COVERS The cloud cover the run was set to (environment.cloud_cover), whatever the rain adds to CARLA's cloudiness; when it named none, CARLA's cloudiness: clear below 20, overcast from 70, partly_cloudy between
road_surface_condition ROAD_SURFACE_CONDITIONS CARLA's water: standing_water from 50 standing water, wet from 10 of either, else dry
illumination ILLUMINATION_LEVELS Sun altitude
rain, fog, wind, cloudiness INTENSITY_LEVELS CARLA weather (0-100): precipitation, fog_density, wind_intensity, cloudiness
road_wetness INTENSITY_LEVELS CARLA's wetness (0-100) alone: the standing water the rain brings is the rain's
tire_friction float, share of the dry road's grip (1.0) The wheel friction the run's weather left the vehicles; nothing when nothing set it
visibility float, m How far drivers see, as the run's weather left it (Visibility follows the conditions); nothing when the run set none
traffic_density TRAFFIC_DENSITY_LEVELS Other vehicles within NEARBY_RADIUS_M
nearby_vehicle_count int Other vehicles within NEARBY_RADIUS_M
pedestrian_nearby bool A walker within NEARBY_RADIUS_M
vehicle_<field>_mean, vehicle_<field>_std, pedestrian_<field>_mean, pedestrian_<field>_std, bicycle_<field>_mean, bicycle_<field>_std float The mean and std of a traffic parameter's spread, as the run set it (Traffic parameters from the ODD); nothing when the run set none

The Lanelet2 probes need the run to have a Lanelet2 map (map.lanelet2_path). Without one they return nothing, except in_junction, junction_type and lane_count, which then read the CARLA waypoint, and speed_limit_kph, which then reads CARLA's get_speed_limit() for the ego.

The probes whose value the map alone decides (speed_limit_kph, lanelet_speed_limit_kph, lanelet_location, lanelet_subtype, road_type, drivable_area_type, in_junction, junction_type, lane_count) can also read it off a lanelet, which is how a planned route is checked before a run (Checking a planned route) and how a sweep picks the road a drawn case is on (The road a case is on).

What they read off a lanelet is public too, for code that has to judge a lanelet exactly as the ODD does: lanelet_in_junction(lanelet) (a turn_direction tag, the default ODD's junction_type), lanelet_road_type(lanelet) (the subtype tag, which the road type and the drivable area type follow) and lanelet_speed_limit(lanelet) (the speed_limit tag, km/h; None for a tag that is not a finite number above 0, as for a missing one). The adversities' location gate reads them (autoware_carla_scenario.adversity.location).

Import them from autoware_carla_scenario.odd itself (from autoware_carla_scenario.odd import rain). The Codon model that checks an ODD has no probes submodule.

Scenario measures

The road users a scenario stages -- a vehicle ahead in the next lane, a pedestrian running out -- are not read by ODD probes: they are measures of the scenario, under keys the framework fixes (autoware_carla_scenario.measures). Every BaseScenario has these, read generically around the ego:

Key Unit Version Reads
vehicle_ahead_gap_m m 2 The free road to the nearest vehicle ahead on the ego's lane or a lane beside it: its rear bumper less the ego's front bumper, along the ego's lane on the Lanelet2 map (up to AHEAD_RANGE_M); a vehicle about to cut in is measured from before it changes lanes
vehicle_ahead_relative_speed_kph km/h 2 That vehicle's speed along the ego's lane less the ego's
crossing_pedestrian_gap_m m 1 How far ahead the nearest pedestrian ahead that is moving was when it set off, held while it crosses
crossing_pedestrian_speed_ms m/s 1 That pedestrian's speed
stop_line_distance_m m 1 Along the ego's lane, from its front bumper to the nearest stop line ahead -- a traffic light's, a traffic sign's, a detection area's or a road marking's, as a stop_line_offset binding finds them -- up to AHEAD_RANGE_M, on any branch of the lane where it forks (the measure does not know the route); nothing without a lane to read along

The pedestrian measures read in the ego's frame: ahead along its heading, within PEDESTRIAN_LATERAL_M of its centre line.

The vehicle ahead

The vehicle-ahead measures read along the road, as a driver would say "the car in front is 15 m ahead", and from the map:

  • The ego's lane is the lanelet the ego is on -- of those under its origin (within ON_LANELET_M), the one whose centreline runs closest to its heading, within EGO_LANE_HEADING_DEG -- with every lanelet following it up to AHEAD_RANGE_M ahead and the lanelets behind it. Where the lane branches, every branch is the ego's lane: the measure does not know the route, so a vehicle ahead on either branch counts. Behind, the lane runs back one way only, through the first of the lanelets before each (Lanelet2's previous(...)[0]), where lanes merge; a lanelet joining the ego's lane is not its lane, nor beside it. Its arc length s runs along their centrelines, on from one to the next.
  • A vehicle counts when its origin is on one of those lanelets, or on a lanelet beside one of them -- its left or right neighbour in the Lanelet2 routing graph, whether a lane change is allowed across the line or not -- and further along the ego's lane than the ego's origin. A lane two over, an oncoming lane or a vehicle off the lanelets does not count, however near.
  • The gap is s of the vehicle's rear bumper less s of the ego's front bumper. A bumper is the middle of the end of the actor's bounding box, along its own heading (Bumpers, bumpers_of), and it is projected onto the ego's lane: onto the lanelet the vehicle is on, or the one it is beside, and those before and after it. So the gap does not depend on where the lanelets beside each other are cut or how long they are -- the outer lane of a bend is longer than the inner one, and the same s on the two is not side by side -- and on a bend it is the distance along the road, not a straight line. A vehicle on the ego's own lane that overlaps it reads a negative gap; one in a lane beside that overlaps it (its rear bumper behind the ego's front bumper) is alongside, not ahead, and does not count, so it never hides the vehicle that is ahead. One beside and ahead of the ego counts, which is how a vehicle about to cut in is measured before it changes lanes.
  • The relative speed is the vehicle's velocity along the lane's tangent at its rear bumper less the ego's along the tangent at its front bumper: two vehicles keeping pace round a bend read 0 though their headings differ.

Without a lane to measure along -- no Lanelet2 map loaded, or the ego on no lanelet that runs its way -- the measures fall back to the geometric definition: a vehicle ahead of the ego's origin within three LANE_HALF_WIDTH_M of its centre line counts, and the gap is bumper to bumper along the ego's heading. The run logs a warning the first time, and the coverage file says how many frames each definition read (measures.vehicle_ahead_gap_m.definitions: along_lane, geometric; vehicle_ahead_definitions() in Python). The geometric definition, too, does not count a vehicle alongside in the next lane.

A reading costs about 3 ms with ten vehicles around the ego on CARLA Town04_Opt (it was some 170 ms). The ego's lane is found once and kept while its lanelet is still the ego's -- the ego on it, running its way, and no lanelet overlapping it (found by geometry, however the map cuts a fork's branches or an exit into lanelets) running closer to the ego's heading -- and the whole map is searched only when it is not (or, off the lanelets, once the ego has moved 2 m or turned 5°); a vehicle is placed by the lane's own lanelets, not by searching the map; and a projection reads only the stretch of a centreline near the point. The Lanelet2 routing graph the lane is read from -- some 25 s to build on Town04_Opt -- is built in the run's setup whenever the ODD maps a vehicle-ahead measure that the scenario reads with the built-in one -- not for one it replaced with register_measure (prepare_measures) -- with every lanelet's bounding box, not on the first tick. A failed reading reads nothing and is warned of once a run.

Measure versions

A measure's version (Measure.version, measure_version(key)) says which definition it reads with, and is raised whenever what it reads changes meaning. The vehicle-ahead measures are version 2. Version 1 (before autoware-carla-scenario 12.0) read the gap centre to centre along the ego's heading -- 4.75 m more than version 2 for a Lincoln MKZ behind a Mini on CARLA 0.10 -- and counted a vehicle within three half-lanes of the ego's centre line, whatever the map.

Coverage keeps the versions apart. openodd's merge treats two coverage items as the same when their name, buckets, event and criteria agree; it does not read their description. So the cover item of an ODD attribute mapped onto a measure past version 1 is written under its version, odd.dynamic.vehicle_ahead_gap@v2: merged with files written before, the report lists odd.dynamic.vehicle_ahead_gap (version 1) and odd.dynamic.vehicle_ahead_gap@v2 (version 2) apart, never one bucket's hits added to the other's. The mark is @v<n>, not #<n>: #<n> is the suffix openodd's merge gives an item it meets defined differently (version 1 files whose buckets changed merge as odd.dynamic.vehicle_ahead_gap#2), and openodd's sampler adds up an item's #<n> definitions. A coverage-driven sweep (strategy: coverage) reads only the current version: OddSampler hands openodd's sampler the @v2 items under the attribute's own item name and leaves the older ones out (current_measure_coverage), so a coverage_from that also holds version 1's files is safe. The 12.0.0 release wrote version 2 as #2; those files read as version 1, and their hits are left out of a coverage-driven sweep on purpose: since 12.0.0 a vehicle alongside in the next lane no longer counts as ahead, which changed what version 2 reads, not only its name. Each coverage file also lists the measures its ODD reads under measures, with their version, unit and text. A scenario that replaces a built-in measure (register_measure) promises the same quantity, so its replacement carries the built-in's version.

Each reads nothing (missing) while there is no such road user. A scenario that knows better replaces one -- it knows which actor is its cut-in vehicle -- or adds one of its own:

class MyCutIn(BaseScenario):
    def setup(self) -> None:
        ...
        self.register_measure(VEHICLE_AHEAD_GAP_M, self._gap_to_my_npc)

An ODD maps its taxonomy onto measures: in Python with scenario_measure(key) as the attribute's probe, in a binding file with measure: in place of probe::

OddAttribute(
    "dynamic.vehicle_ahead_gap",
    scenario_measure(VEHICLE_AHEAD_GAP_M),
    unit="m",
    buckets=[0, 10, 20, 30, 50, 100],
)
probes:
  headway: {measure: vehicle_ahead_gap_m, buckets: [0, 10, 20, 30]}

The attribute then reads the running scenario's measure, its replacement included, in the measure's unit: an attribute mapped onto a built-in measure takes its unit, and one given that is not it is refused. The keys are in the Codon model (from autoware_carla_scenario.measures import VEHICLE_AHEAD_GAP_M), so a scenario or a Python ODD naming them is checked like any other (Static Check): autoware_carla_scenario.measures is modelled whole, Measure, BUILT_IN_MEASURES, read_measure(), check_measure_key() and measured_scenario() included. A Measure is frozen, as a Metric is: assigning to one of its fields fails the check, as it fails the run. A scenario may build its own (Measure("my_measure", read, "m")); the check calls its read with a world and refuses one that returns anything but a float or an Optional[float]. A metric's keys are modelled the same way (Metrics): autoware_carla_scenario.metrics is modelled whole, Metric and BUILT_IN_METRICS included. The ODD names a measure key, never a scenario; the scenario names its measures, never the ODD. The default ODD maps none: what a scenario sets up is no condition of the domain, but an effect the rules draw (openodd's docs/rules.md, "Effects a scenario sets"), and a case records the value it drew (case.quantities), which the logical parameter space is plotted over.

Measures and metrics

A measure reads a condition the ego drives in: what the scenario sets up. How the ego drives -- whether it collides, leaves the road, drives against the traffic, how close it comes to other road users -- is an outcome, read by a metric (Metrics). An ODD cannot map onto a metric: scenario_measure("offroad"), or measure: offroad in a binding file, is refused. Were it allowed, the ego's own driving would decide whether a run is inside the domain it is tested in, and an ODD excluding offroad would report a run that left the road as outside the ODD rather than as failed. A run is judged on a metric with MetricCondition instead, once it is over.

Writing an ODD in OpenODD YAML

The OpenODD files are plain OpenODD 1.0. The example below follows the standard's own examples (Code 170, 171 and 192):

# taxonomy.yml
TAXONOMY:
    environment_conditions:
        rainfall_rate: float precipitation_rate
        rainfall_level:                       # literals defined by ranges: ordered
            no_rain:
                rainfall_rate: "< 0.1 mm/h"
            light_rain:
                rainfall_rate: "[0.1 .. 2.5] mm/h"
            heavy_rain:
                rainfall_rate: "> 2.5 mm/h"
        wind_speed: float velocity
    scenery:
        road_type: [road, highway, road_shoulder]
        lane_count: integer count
# odd.yml
IMPORT:
    - taxonomy.yml

ODD:
    odd1:
        TITLE: The baseline ODD
        INCLUDE_AND:
            low_speed_roads: true
            needs_wind: true
        EXCLUDE_OR:
            bad_weather: true

MODULES:
    low_speed_roads:
        INCLUDE_AND:
            road_type: [road]
            lane_count: "< 3"
    bad_weather_1:
        LABEL: bad_weather
        INCLUDE_OR:
            rainfall_level: ">= heavy_rain"
            wind_speed: "> 50 km/h"
    needs_wind:
        EXCLUDE_OR:
            wind_speed: unknown               # the wind speed is required

The binding file is this framework's own. It says which probe measures each concept, and its buckets:

# urban.yaml
openodd: [odd.yml]                 # relative to this file
name: urban                        # default: this file's stem
text: Urban roads, fair weather
probes:
  road_type: {probe: lanelet_subtype}
  rainfall_rate: {probe: my_package.probes:rain_rate_mm_h, unit: mm/h}
  wind_speed: {probe: my_package.probes:wind_mps, unit: m/s}
  lane_count: {probe: lane_count, values: [1, 2, 3, 4], cover_by: meters, target: 200, min_stay: 2}

Run with odd=path/to/urban.yaml. An OpenODD file on its own can be named too, but then nothing is measured: every concept is missing.

Taxonomies and modules from git

An openodd entry can name a file in a git repository at a revision, so a taxonomy or a module library kept elsewhere is used as it was at that revision:

# urban.yaml
openodd:
  - git: https://example.com/odd/taxonomy.git
    rev: 'v1.2.0'                  # a tag, a branch, or a commit
    path: taxonomy.yml             # a list for several files
  - git: git@example.com:odd/modules.git
    rev: '3f2a9c1e0b7d4a6c8e5f1a2b3c4d5e6f7a8b9c0d'
    path: odd/urban.yml
  - local_overrides.yml            # relative to this file
probes:
  ...
  • An IMPORT is looked for next to the importing file first, and then in the other sources: the directory of each file named, and the root of each repository. The standard makes file names unique within one transmission, so odd/urban.yml in one repository can IMPORT: [taxonomy.yml] from another. A name found in two sources is refused.
  • git fetch does the fetching, so url is anything it takes, with the credentials it is configured with. Prompts are turned off, and the ext:: transport is refused.
  • Repositories and checkouts are cached in $AUTOWARE_CARLA_SCENARIO_ODD_CACHE when it is set, else openodd's $OPENODD_CACHE (default ~/.cache/openodd). A full commit id checked out once is read from the cache without the network; a tag or a branch is fetched on every load, to learn which commit it names now. Pin a commit for runs that must be reproducible.
  • Quote rev: unquoted, a commit id of digits and one e (1630255e36) or a long one of digits only is read as a number, and looked for as one (1.630255e+36).
  • Fetching needs git, which the scenario image does not have. A run there can still read a source pinned to a full commit id, from a cache filled beforehand (scenario-odd show urban.yaml with the same cache directory) and mounted at $AUTOWARE_CARLA_SCENARIO_ODD_CACHE.
  • The commit each source was read at is written to the coverage file (the ODD's sources), and scenario-coverage lists it, warning when the merged runs read different commits of a source.

From Python, pass GitSource(url, rev, path) to load_openodd().

What is read

IMPORT: other files, relative to the importing one, or else in the other sources (see above). A cycle is refused.

Every file read with an ODD makes up one transmission, in the standard's words, and the standard makes two things unique within one. Both are refused rather than guessed at:

  • two different files of the same name, wherever they come from;
  • a concept defined in two files, even the same way. A file can still add concepts to a container another file started (weather: in both, with different concepts under it).

TAXONOMY: nested mappings are records and containers. A leaf is one of:

  • integer|long|float|double <unit type>: a number;
  • boolean;
  • a list of literals: a categorical;
  • a mapping of literals to expressions on other concepts: a categorical defined by them. Its value is the literal whose expressions hold; on a shared range endpoint, the literal written first wins. Defined by ranges, its literals are ordered in the order written, so "< heavy_rain" means the literals before it;
  • the id of a categorical: the same literals.

A reference to a record (a user-defined type) and a shapefile are not followed.

MODULES and ODD: modules. Those under ODD are the root candidates: the ones among them no other module refers to are the roots. The standard's examples also put referenced modules there. Without an ODD section, every module no other module refers to is a root.

A module's keys are:

  • TITLE, DESCRIPTION;
  • ACTIVE;
  • LABEL / LABELS (one name or a list);
  • METADATA (ignored, here and inside a section);
  • one INCLUDE_AND or INCLUDE_OR, and one EXCLUDE_AND or EXCLUDE_OR.

A section nests one level of the other operator: OR: in an AND section, AND: in an OR one.

A section maps a concept to an expression, or a module or label to true/false. A concept is named by its id (wind_speed), or by as much of its path as makes it unique.

Expression Meaning
low, 3, true, "0 mm/h" equal
[a, b] one of the literals
"> x unit", ">= x", "< x", "<= x" a bound (> excludes x, >= includes it)
"[low .. high] unit", "[low, high] unit" an inclusive range
"< literal", "[a .. b]" a bound or range over an ordered categorical
unknown (none, null, undefined, or YAML null) the value is missing, unless the categorical has a literal of that name

A number's unit must measure the concept's unit type. It is converted into the probe's unit, so "> 50 km/h" against a probe in m/s is > 13.89. The units known are listed below; a document adds more with OpenODD's conversion: block.

Unit type Units
length m, km, cm, mm, mi, in, ft
velocity m/s, km/h (kph), mph
acceleration m/s^2, g
time s, ms (msec), min, h
angle deg, rad
precipitation_rate mm/h (mm/hr)
temperature C, K, F
fraction %
bandwidth bps, kbps, Mbps, Gbps
frequency Hz, kHz
illuminance lx
count, occurrence count, 1/h, occ/hr

The following are refused, each with the module it appears in:

  • a concept that is not in the taxonomy;
  • a literal a categorical does not have;
  • a unit of the wrong type;
  • a second include or exclude section;
  • deeper nesting;
  • a module or label id containing unknown.

The YAML is read with YAML 1.2 booleans: only true and false are booleans, so literals such as off or NO stay literals. A key written twice is refused. As the standard notes, an unquoted expression starting with > (speed: > 50 km/h) is a YAML block scalar, so quote it.

Not read: COD / OD records, numeric terms (1.75*ego_width), $ parameters, and condition-level metadata.

The binding file

Key Meaning
openodd The OpenODD files, relative to the binding file
name, text The ODD's name and description
probes Concept → probe (built-in name or package.module:function) or measure (a scenario measure's key, see Scenario measures), unit (built-in probes and measures know theirs), buckets (values, buckets, or range + every), text, and the coverage criteria target, cover_by, min_stay
situations Module → its criteria (target, cover_by, min_stay, all optional): the modules to cover as situations (see Situations above)
situations:
  three_lane_cruise: {cover_by: meters, target: 200, min_stay: 3}
  urban_junction_rain: {}          # one stay of any length

Situations live in the binding file, not in the OpenODD documents: OpenODD has no notion of coverage, and a construct of its own in an OpenODD document would not be compliant. A module named under situations and not under the ODD section is not a root candidate.

When no buckets are given:

  • a categorical gets one bucket per literal, and a boolean gets two;
  • a number gets buckets at the thresholds the modules test it against. "<= 60 km/h" makes [-inf, 60) and [60, inf], so every boundary the ODD draws is tested from both sides. Each bucket holds its lower edge, so [60, inf] holds 60, which <= 60 lets in. That bucket is partly inside, so it stays a coverage target. With "< 60 km/h", it would be outside.

A categorical defined by expressions is measured when what its expressions read is. A concept with no probe is always missing, and is reported as monitored but not covered.

Picking the ODD of a run

The odd config key, or ScenarioQueue(odd=...) / ScenarioRunner(odd=...), takes one of:

Value ODD
default The built-in one (Coverage): every attribute, no modules. It is openodd's bundled default ODD (its files), measured by this framework's probes, so it has a revision like any ODD read from files
a name One registered with register_odd(name, builder) in code, or by a package through the autoware_carla_scenario.odds entry point group (or openodd's openodd.odds)
path/to/urban.yaml A binding file (or an OpenODD file on its own, measuring nothing)
package.module:function A builder to import and call
uv run scenario scenario=intersection_passing/left_turn odd=path/to/urban.yaml
uv run scenario scenario=intersection_passing/left_turn odd=my_package.odds:urban_odd

A package that ships its ODD registers it through the autoware_carla_scenario.odds entry point group, or openodd's openodd.odds when the ODD is not CARLA's alone (both are read; openodd.resolve_odd() resolves names as resolve_odd() does, with files measured by its backend). The entry point names a function that takes nothing and calls register_odd, the convention the scenario and traffic-backend groups use. A private taxonomy can then plug in without the framework knowing it:

[project.entry-points."autoware_carla_scenario.odds"]
my_package = "my_package.odds:register"
def register() -> None:
    register_odd("urban", urban_odd)

Sampling scenarios from the ODD

Coverage says which situations of the ODD the runs so far have driven. The sampler draws the settings of the next runs from the ODD: only combinations the ODD admits, and, if asked, aimed at what is still uncovered. It is a sweep, so it is run by scenario-expand and by the lanelet sweeper, and with autoware.launcher each case gets a fresh Autoware (Autoware as the Ego):

uv run scenario --multirun hydra/sweeper=lanelet_constraint \
  scenario=intersection_passing/left_turn map=town10hd_opt \
  +sweep.odd_sample.count=20 +sweep.odd_sample.strategy=coverage \
  '+sweep.odd_sample.coverage_from=[outputs]'

A glob batch with --multirun draws each matched scenario's cases the same way and runs them all in one queue (Batch Execution). Scenarios that draw from one ODD go on from each other's draws, in the order the batch lists them: with strategy: coverage a later scenario counts the earlier ones' cases as covered and aims at the buckets and situations they left (see How a case is drawn); with strategy: uniform it draws from a random stream of its own rather than repeat their values. Here cut_in/right's 3 cases aim at what cut_in/left's 3 left uncovered:

uv run scenario --multirun 'scenario=cut_in/*' map=town10hd_opt \
  +sweep.odd_sample.count=3 +sweep.odd_sample.strategy=coverage

scenario-expand on the same glob lists those same cases (without the arguments the batch adds), each scenario going on from the earlier ones' draws, as in the batch.

The same seed, ODD, knobs, coverage and scenarios, in the same order, give the same cases, on any machine (openodd's sampler draws from ChaCha8 seeded with the seed's SHA-256). Two ODDs are one when they are built alike (their structure), however the command line names them (a registered name, a YAML path); a scenario on another ODD starts afresh.

or in the scenario's YAML:

sweep:
  odd_sample:
    count: 20              # cases, a whole number >= 1; one per lanelet case when left out
    seed: null             # a whole number in 0..2147483647 (2**31 - 1); the run's (`seed`) when null, `DEFAULT_SEED` (0) when the run has none either; same seed, ODD, knobs and coverage: same cases
    strategy: coverage     # uniform | coverage | weighted (weighted when `weights` is given)
    coverage_from: [outputs]   # earlier runs' *_coverage.json (files or directories)
    weights: null          # module -> share of the cases, or a YAML file of them; see "Sampling by shares of the ODD"
    odd: null              # the ODD to draw from; the run's (`odd`) by default
    knobs: {}              # by ODD attribute, replacing the scenario's controls; see below
    flaky: null            # look for flaky cases; see "Finding flaky cases" below
    tire_friction: true    # draw each case's grip in the range its weather allows; see "Tire friction follows the weather"
    visibility: true       # draw how far each case's drivers see in the range its weather allows; see "Visibility follows the weather"

count and seed are whole numbers (20 or 20.0, not 2.7 or a boolean), count at least 1 and seed from 0 to 2147483647 (2**31 - 1); any other value is refused, naming the key, before the map is read or anything starts. So are an odd or strategy that is not a name (false, 0, "") and knobs that are not a mapping (false, a list): a field left out or null takes its default, any other value is refused rather than read as none.

Each case is a list of Hydra overrides. A sweep that also has constraints (or a route search) gives each drawn case one of its lanelet cases -- in turn unless the map decides some of the ODD's attributes, as it does the default ODD's junction, lane count and speed limit: then each drawn case goes on a lanelet case in the buckets it drew, so a lanelet case may run more than once or not at all (The road a case is on). One without draws from the ODD alone, and needs no map to expand. The last override, ++odd_case=..., is the case itself, encoded: the run records it, with its outcome, in its coverage file (see Finding flaky cases).

A drawn case's settings are the sweep's to choose, so the run's own overrides may not set them: environment.time_of_day=night on the command line of a sweep that draws the time of day is refused, naming the key and the ODD attribute that draws it, before any case runs -- by scenario-expand, a --multirun batch (local or submitted to SLURM), the lanelet sweeper and the UI's Run panel alike. Left to Hydra, the override would win at run time while the case recorded the label drawn, its sun and other weather effects followed that label, coverage counted its bucket and the flaky search learnt from a value the run never had. To hold an attribute, give it a knob in the sweep that draws only what you want -- knobs: {<attribute>: {key: environment.time_of_day, values: {night: night}}} for one label, a narrower range for a number -- or narrow the ODD; then the case, its weather and its coverage follow it. A CARLA weather field the conditions lead to (the sun's altitude, the precipitation) and a scenario control's node are refused the same way: change the rules they follow instead. Overrides of keys no case sets (map=, ego.entity=) are not concerned, and one that names the grip or the visibility, or where in its range it is, turns that draw off, as a case that names one keeps it (Tire friction follows the weather, Visibility follows the conditions).

Who knows what

Three things meet in a sampled case, and none of them knows the others:

Says Knows
The ODD which conditions the system is for, and which scenario measure each of its attributes is (its mapping) its taxonomy and the measure keys; no scenario
A scenario its measures (BaseScenario's, or its own replacements) and its controls: which of its parameters sets which measure, and what it can stage the measure keys; not the ODD's taxonomy or bounds
The framework the measure keys, and the weather and sun every scenario can be given neither

An ODD tightened, loosened or written in another taxonomy needs no scenario changed; a scenario added needs no ODD changed. What a scenario sets up is not a condition of the domain but an effect: the nodes it adds to the rules (A scenario's nodes), drawn with every case whatever the ODD names. A control without a quantity is still joined to the ODD on the measure key: an attribute the ODD maps onto measure k is drawn with the running scenario's control of k, and controls of measures the ODD does not map are unused. A control with a quantity is never an attribute's knob, in a Python ODD or a binding file alike (CarlaBackend.knob): an ODD that maps an attribute onto its measure (headway: {measure: vehicle_ahead_gap_m, buckets: [0, 10, 20, 40]} beside cut_in/left) draws the control as the scenario's effect, and the attribute measures what it drew; a case's settings file the control's key under the control, not the attribute (#2072). A sweep's knob for that attribute (sweep.odd_sample.knobs) replaces the control, as an explicit knob replaces any control: the attribute is drawn with the knob, and the control is not drawn as an effect. Attributes nothing sets are left open: a combination is admitted when the ODD can hold for some value of them.

A scenario's nodes

A scenario adds nodes of its own to the shared rules (the weather rules): what it sets up -- the gap to a vehicle about to cut in, how fast a pedestrian crosses -- as a rule document under scenario_rules in its config, version 3, joined to the shared network (openodd's Backend.add_rules, its docs/rules.md "A scenario's nodes"). A control that names a node by quantity writes it; a sweep draws every node with every case, at a quantile of its own the flaky search can move, and the case records the value it drew (case.quantities), which openodd's case plot places the runs by:

# scenario/cut_in/left.yaml
scenario_rules:
  version: 3
  taxonomy: {bundled: default}
  basis: {kind: expert, by: autoware_carla_scenario}
  quantities:
    vehicle_ahead_gap:
      parts:
        staged:
          uniform: {low: 2.0, high: 30.0}      # what the scenario can stage
    vehicle_ahead_relative_speed:
      parts:
        staged:
          uniform: {low: -15.0, high: 15.0}
controls:
  vehicle_ahead_gap_m:
    key: scenario.npc_gap_m
    target: measure                            # the gap the measure reads at the start
    quantity: vehicle_ahead_gap
  vehicle_ahead_relative_speed_kph:
    key: scenario.npc_initial_speed_kmh
    quantity: vehicle_ahead_relative_speed
    offset: ${ego.initial_speed_kmh}           # its speed less the ego's
    limits: [0.0, null]                        # a speed: never below zero

A node without at is a root: nothing in the domain conditions it yet. One may read the shared nodes -- a gap that lengthens on a wet road reads the grip -- and may not redefine one: the join refuses a node both define, so a scenario can never change what another's cases follow. Each scenario's nodes are its own: the cases of one never draw another's. Nor may a scenario name a node tire_friction_quantile or visibility_quantile: a sweep adds those to its nodes to draw where each case's grip and visibility are (see Tire friction follows the weather).

What the run staged. A control is keyed by the measure it sets, so that measure reads, in the run, the node the case drew: a run of a drawn case measures each node its case drew by its control's measure -- the vehicle_ahead_gap by vehicle_ahead_gap_m, the crossing_pedestrian_speed by crossing_pedestrian_speed_ms -- on every tick of its first 10 s, beside the ODD and counting toward no grade (openodd's CoverageCollector(effects=...), written as effects.samples in the coverage file). openodd's drift view then sets what the case drew against what the run staged, run by run: a gap placed 2 m off, or a relative speed the ego's first tick of acceleration moved, shows as drift, and its correction is the control's scale and offset (openodd's docs/drift.md, "Effects: what a scenario sets up"). A run no sweep drew measures none.

Controls

A control (an OddKnob) ties a measure to the config key that sets it. A scenario declares its own in its config, next to the parameters they set, keyed by measure (A scenario's nodes has the example scenarios' own).

Field Meaning
key The config key the value is written to
quantity A node of the rules (the scenario's scenario_rules, or the shared network's) whose value is written, drawn with every case: the control is then the scenario's effect, not an ODD attribute's knob, and range and values are not used
target setting (the default, as is null): the key is a raw setting, written through scale and offset. measure: the key takes the measure's value at the start itself, written unchanged -- no scale, offset, integer or values (see below)
range Numeric: [low, high] in the measure's unit, the values the scenario can bring about. A bucket is drawn only where it meets the range (by more than an edge), and only that part of it
values Per bucket label. Categorical: the value to write, or [low, high) to draw it from ([x, x] is x); a bucket without one is not drawn. Numeric: a range to draw from instead of the bucket's interval -- which an unbounded bucket (-inf/inf edges) needs to be drawn at all
scale, offset A numeric value drawn in the unit is written as value * scale + offset (scale not zero)
integer Write an integer. A rounded numeric value that falls in another bucket is drawn again
limits Numeric: [low, high] the written value must lie in, null for an open end -- what the key takes, whatever offset comes to. A bucket only values outside it would reach is not drawn

Checking controls

scenario-check validates controls: of every config it checks, and scenario-expand, a sweep and a --multirun batch refuse invalid ones -- a concrete scenario's, without a sweep:, as well as a logical one's -- each problem as one line with its config path, not a traceback:

cut_in/left: controls.vehicle_ahead_gap_m.range: expected [low, high] with low <= high, got [30, 2]
cut_in/left: controls.vehicle_ahead_gap_m.rnge: unknown field (did you mean 'range'?)
cut_in/left: controls.vehicle_ahead_gap_m: expected a key or a mapping, got 5
cut_in/left: controls.vehicle_ahead_gap_m.key: 'scenario.npc_gapm' is not a key of the config (did you mean 'scenario.npc_gap_m'?)

Errors (exit status 1; the config is not compiled): a control that is not a key or a mapping, an unknown field, a missing or unknown key (it must be a key of the composed config), a target that is neither setting nor measure, scale/offset/integer/values under target: measure, a range or limits that is not [low, high] with low <= high, a quantity that is not a name, an integer that is not true or false ("false" and 0 are refused, not read as true and false), values that are not a mapping, a scale or offset that is not a number. A field given null is one not given: its default.

The knobs of sweep.odd_sample.knobs are checked the same way, before the ODD is resolved, each problem naming its full path (sweep.odd_sample.knobs.<attribute>.<field>: ...) in a ValueError; so are the shares of an inline weights (#2226).

A measure name that is not a built-in one is only a warning (controls.vehicle_ahead_gap: not a built-in measure (did you mean 'vehicle_ahead_gap_m'?)): a scenario may register measures of its own at run time (register_measure), which a check of the config cannot see, so it cannot tell a typo from such a measure.

{measure: key} is short for {measure: {key: key}}. The ODD's attribute and the measure share a unit: the binding takes the measure's.

Raw settings and measure-targeted controls

A raw setting (target: setting) is a scenario parameter the drawn value is mapped onto by scale and offset, and the mapping is the author's claim that the parameter brings the measure about. It holds where the two are the same quantity by definition. It is a guess where they are not: npc_ahead_m is the NPC's s along its lanelet less the ego's along its own, while vehicle_ahead_gap_m is bumper to bumper along the ego's lane. Neighbouring lanelets do not run alike on a curve -- the outer one is longer, and their s do not line up -- and the bumpers depend on the vehicles, so no constant offset is right wherever the scenario is placed (on CARLA 0.10 Town04_Opt one guessed +11 m was measured 2.2 m off, and the right offset there is right nowhere else).

A measure that the initial placement alone determines can be the parameter instead. The scenario takes "the measure should read v at the start" and solves for the placement that makes it so, evaluating the measure's own definition on the poses it will spawn at; the control says target: measure, and the value drawn is written unchanged:

from autoware_carla_scenario import place_at_vehicle_ahead_gap, vehicle_bumpers

# The NPC's s on its lanelet at which vehicle_ahead_gap_m reads cfg.npc_gap_m,
# seen from where the ego spawns -- whatever the two lanelets' shapes.
ego_at = to_carla_world(self._spawn_pose).to_carla_transform()
npc_pose = place_at_vehicle_ahead_gap(
    ego_at,
    cfg.npc_lanelet_id,
    cfg.npc_gap_m,
    ego_bumpers=vehicle_bumpers(world, self.ego_config.vehicle_type),
    npc_bumpers=vehicle_bumpers(world, cfg.npc_vehicle_type),
)

place_at_vehicle_ahead_gap reads the gap with the measure's own code (measures.vehicle_ahead_gap_between, which the measure reads every vehicle with) for the NPC at each s of its lanelet, as a Lanelet2Pose spawns there -- on the centreline, heading along it -- and narrows the first crossing of v down by bisection (solve_along, a 1-D root find that takes any s -> measure function, for a measure of a scenario's own). It solves on the NPC's own lanelet and projects the NPC onto the ego's lane, so it does not matter where the lanelets beside each other start or how long they are. A gap the lanelet cannot reach -- too short, turning away, or out of the lanes the measure counts -- is refused with the gaps it does reach, so a drawn case that cannot be staged fails at setup rather than running another gap.

The gap is bumper to bumper, so the placement needs both vehicles' sizes before either is spawned. CARLA gives a blueprint no size: vehicle_bumpers reads the bounding box of a vehicle of the blueprint already in the world when there is one, and otherwise spawns one high above the map (higher again if that spawn is refused), reads its bounding box and destroys it before the world ticks again -- once per blueprint per process. The vehicle-ahead measures do not count a vehicle that far above the ego. The NPC is then spawned once, where the gap reads right, rather than spawned, measured and moved -- a move that would race its initial speed and the first tick.

Which to use:

The measure is Control
decided by where and how road users start, and the parameter is that measure by definition (npc_initial_speed_kmh less the ego's is the relative speed) a raw setting, with the offset that makes it the measure
decided by where road users start, but the parameter is another quantity (an arc length for a gap) target: measure, with the scenario solving its placement
decided by what the run then drives (a pedestrian's gap when it sets off, anything a controller or traffic changes) a raw setting; how far the measure drifts from what was drawn is read off the runs, in openodd's case plot (its drift/calibration view), not solved at setup

The relative speed of cut_in is the first kind: both vehicles start at their speeds along their lanes, and the measure is the NPC's velocity along the ego's lane at the NPC less the ego's along it at the ego, so its offset: ${ego.initial_speed_kmh} writes it exactly: on a bend the NPC heads along its lane, which runs alongside the ego's lane where it is. (A lane beside a bend is shorter or longer than the ego's, so an NPC on it covers the ego's lane a few per cent slower or faster -- 3.5 m off a 100 m radius, 3.5 % -- which the measure does not count: it reads speeds, not how fast s changes.)

The weather and the sun are set for every scenario, through the environment config, which sets them before the run starts (EnvironmentAction). The default ODD names the conditions -- the rainfall rate, the visibility through the fog, the time of day, the wind speed and the cloud cover, the taxonomy's -- and the built-in probes reading them have controls of their own:

Probe Key Range
rainfall_rate_mm_h environment.rainfall_rate_mm_h 0-30 mm/h, within the bucket drawn (the rain intensity's: light 0.1-2.5, moderate 2.5-7.6, heavy above)
fog_visibility_range_m environment.fog_visibility_range_m 15-5000 m, within the bucket drawn (the fog level's: thick under 50, heavy 50-450, medium 450-800, mist 800-1600)
time_of_day environment.time_of_day the label drawn: day, dusk, night, dawn
wind_speed_kmh environment.wind_speed_kmh 0-100 km/h, within the bucket drawn (the wind intensity's: light 5-20, moderate 20-50, strong above)
cloud_cover environment.cloud_cover the label drawn: clear, partly_cloudy, overcast

What CARLA cannot bring about -- snow and the temperature, which the default ODD's taxonomy declares -- the CARLA backend names (CarlaBackend.unsupported, odd.backend.UNSUPPORTED_CONCEPTS): an ODD that measures one, or whose modules restrict one or a concept defined by one (the snow intensity), is refused as it is read, before anything runs (openodd.UnsupportedConceptError), rather than covered by runs that never rendered it. The default ODD declares them and measures neither.

CARLA does not take these: it renders them by its own weather fields, and which fields render a condition is what the weather rules say -- a Bayesian network from the conditions to CARLA's precipitation, cloudiness, precipitation_deposits, wetness, fog_density, sun_altitude_angle and sun_azimuth_angle, and on to the grip and the visibility. They are worked out once, before the run, and the run applies numbers: a sweep.odd_sample draws each field that follows from a condition the ODD names with the case, at a quantile of its own (an effect, environment.precipitation=... among the case's overrides, its value in the case's quantities, which the flaky search moves); a config that names a condition gets the rest at its median (weather_rules.resolve_environment). A CARLA field the config sets itself is kept, and what follows from it in the rules reads it. The probes read the conditions back: the rainfall rate, the fog visibility, the wind speed and the cloud cover as the run was set to them, the time of day off the sun. The cloud cover is the sky's: rain clouds CARLA's sky past it (the rules' carla_cloudiness is the cloudier of the sky and the rain's clouds), so a case drawn clear in moderate rain renders a cloudy sky and is measured clear, the cell it was drawn for. A run that names no cloud cover is measured off CARLA's cloudiness.

A config that sets CARLA's fields alone names no condition: the ODD's weather is not measured, and drivers see as they always did (a warning says so). CARLA's own fields have controls too, for an ODD that names them:

Probe Key Ranges
rain environment.precipitation none 0, light 1-30, moderate 30-70, heavy 70-100
fog environment.fog_density as rain
illumination environment.sun_altitude_angle day 15-90, low_sun 0-15, twilight -6-0, night -90 to -6
visibility environment.visibility_m 15-400 m, whatever the weather
road_wetness environment.wetness as rain
tire_friction environment.tire_friction 0.3-1.1 of the dry road's grip, whatever the weather
wind environment.wind_intensity as rain
cloudiness environment.cloudiness as rain

The precipitation brings, by the rules, the cloudiness, standing water and wetness of CARLA's rain presets: precipitation alone draws only the falling drops, and a camera sees a dry road under a clear sky. road_wetness reads the weather's wetness alone, which its knob sets, so a drawn value reads back as its bucket; the grip follows the wetter of it and the standing water. tire_friction reads the grip the run's weather left the vehicles' wheels (a share of the dry road's, 1.0) -- None when nothing set it -- and its knob names it outright, as visibility's names the visibility. wind and cloudiness read CARLA's 0-100 intensities as levels, as rain and fog do.

An ODD that names a condition and the CARLA field that follows from it -- the taxonomy's rainfall_rate beside rain or road_wetness, the cloud cover (cloud_cover) beside cloudiness, wind_speed beside wind, fog_visibility_range beside fog -- has the field drawn by the field's knob, not by the rules: the knob wins, and no effect writes that field or any field that follows from it in the rules (the precipitation's cloudiness, standing water and wetness), so every bucket a case records is the one the run reads back (#2072). The runner works those fields out from the value the knob wrote, at their median (weather_rules.resolve_environment, and EnvironmentAction's road water, in which a field set wins), as it does for a config that sets the field itself; the case's quantities and the flaky search hold only the effects left. illumination beside time_of_day is refused when the cases are drawn: its knob sets the sun's altitude, which the time of day is read back from, so no run could keep the buckets drawn for both (ValueError: ODD ...: environment.illumination sets environment.sun_altitude_angle with its knob, which ...time_of_day is read back from ...); an ODD names one of them.

Every environment.* value must be a finite number -- an integer or a float, not a boolean or a string (true and "15" are refused, not read as 1 and 15) -- but time_of_day, one of day, dusk, night, dawn, and cloud_cover, one of clear, partly_cloudy, overcast: any other value, NaN or infinity is refused before the run starts, naming the key (environment.rainfall_rate_mm_h must be finite, got nan). cloudiness, precipitation, precipitation_deposits, wetness, wind_intensity and fog_density must be within 0-100, tire_friction, visibility_m and fog_visibility_range_m positive, fog_distance, rainfall_rate_mm_h and wind_speed_kmh not negative, sun_altitude_angle within -90 to 90 degrees and sun_azimuth_angle within 0 to 360 (CARLA's ranges; a value outside is refused, not wrapped or clamped), and the two quantiles within 0-1. EnvironmentAction refuses the same values with a ValueError naming its argument.

Fog is drawn as fog_density alone. How thick that looks is the server's: a CARLA 0.10 server on the UE 5.8 engine (the ue58-dev tree) multiplies it by carla.Weather.FogDensityScale, whose default 0.001 leaves even fog_density=100 a faint haze on the horizon. Start it with -ExecCmds="carla.Weather.FogDensityScale 0.01" for fog a camera sees, or set server.fog_density_scale=0.01 (LiDAR).

The precipitation and the fog density set the LiDAR noise as well: each driver.lidars sweep carries the world's precipitation and fog density (LiDAR).

A scenario that sets the weather itself has the last word over the config: its own EnvironmentAction, with numbers -- worked out from conditions with EnvironmentAction(**resolve_environment({"rainfall_rate_mm_h": 5.0})).

Tire friction follows the weather

CARLA draws rain but does not drive on it: a vehicle grips a soaked road as it grips a dry one. So an EnvironmentAction that sets the rain or the road's water -- wetness or precipitation_deposits, which rain brings -- also scales every vehicle's wheel friction (friction_force_multiplier) to a share of the dry road's grip, within the range the resulting water leaves (tire_friction_range). The default weather rules give it by the wetter of wetness and precipitation_deposits:

Road water Grip, share of the dry road's The rainfall that leaves it
0 (dry) 0.90-1.10 none
50 (wet) 0.65-0.85 moderate, about 4 mm/h (standing water about 50)
100 (flooded) 0.45-0.65 heavy, 25 mm/h and above

Between the rows the bounds are interpolated (the rules write the distribution at these points). Measured tire-road friction is about 0.8-1.0 on dry asphalt, 0.5-0.7 on wet and 0.3-0.5 under a water film; each range spans what tires, their wear and the asphalt make of the same water, and a dry road's is centred on CARLA's own grip, so a dry run drives as it did before.

Rain brings the water with it however it is set. An EnvironmentAction that sets precipitation and not wetness or precipitation_deposits -- a scenario's own, or the editor's "Set Weather" card -- sets those to what the rules give that rain, at their median (weather_rules.road_water), as the config's environment does before the run: precipitation=100 leaves a wetness of 97.5, standing water of 90 and a grip of 0.56 either way. Rules whose water reads a condition read the run's: the rainfall rate, fog visibility, wind speed or cloud cover the action names, else the one the run was set to (the config's environment, or an earlier action), and the time of day the sun renders once the action has set it. A wetness or precipitation_deposits the action gives wins.

So stopping the rain dries the road at once: under the default rules, EnvironmentAction(precipitation=0) sets wetness and precipitation_deposits to 0 and the grip back to the dry road's, as a config's precipitation: 0 does. CARLA does not let a road dry over time; a scenario whose road should stay wet after the rain gives the water itself, EnvironmentAction(precipitation=0, wetness=60, precipitation_deposits=30).

Besides the water the rain brings (above), the grip is the one quantity the action reads off the rules during the run: it may differ by vehicle (Rules by vehicle), and a vehicle spawned later gets its own. It reads every CARLA field of the weather the action leaves -- the water the run's conditions gave before it started -- and the conditions themselves, as the ODD's probes read them: the rainfall rate, the visibility through the fog and the wind speed the run was set to, the cloud cover it was set to (else the one CARLA's clouds render), the time of day CARLA's sun renders, and how far drivers see. So a tire_friction part may read a cause or any other quantity; a rainfall rate, fog visibility or wind speed the run does not name reads its default, as it does before the run, while the time of day and the cloudiness are always read off CARLA's sun and clouds (the cloud cover the run names wins).

Under rules other than the default ones -- a rule file of the run's (environment.rules) or rules by vehicle -- the grip may differ from CARLA's on any road, so every EnvironmentAction sets it, whatever else it sets, and the config's environment registers one even when it names only rules. One that names no grip, quantile or water (a scenario's own that sets the sun, say) reads the grip again at the quantile the run's grip is at, and keeps a grip named outright (tire_friction). A grip named outright also ends the quantile: water set after it reads the grip at the median. Under the default rules, a run that sets no road water, no quantile and no grip keeps CARLA's grip (1.0, whatever water CARLA renders).

Where in the range a run's grip is, is environment.tire_friction_quantile: the quantile of the grip's distribution (0 its low end, 1 its high end; the median when not set). A later action in the run that names none -- rain starting mid-run -- reads the grip at the same quantile. A sweep.odd_sample draws it for every case (environment.tire_friction_quantile=0.412 among the case's overrides, recorded in its coverage file) as a parameter of the case, quantile(tire_friction_quantile) among its values -- a root node of the rules, uniform on [0, 1], drawn as the weather effects' quantiles are -- so the same sweep draws the same grip, and the flaky search sees and moves it like any other parameter. The weather comes from the ODD and the grip from the range that weather allows: a heavy-rain case never runs on a dry road's grip. tire_friction: false in the odd_sample leaves every case at the median -- of the rules' grip where the case sets it, as above; the dry road's, CARLA's own, where it does not -- (true or false, or one of their spellings, as the run's switches take: "off", "no" and 0 are false; a missing or null one is true; anything else, such as ture or 2, is refused, naming sweep.odd_sample.tire_friction, before any case is drawn), and environment.tire_friction fixes the grip whatever the weather (it wins over the quantile, and a case that names one draws no quantile; the ODD's tire_friction probe draws it through that knob). A run whose own overrides name the grip or its quantile draws none either.

The vehicles in the world when the action fires are scaled then; a scenario with such an action -- the config's, or its own -- also runs follow_tire_friction after every tick, which gives the vehicles spawned later (background traffic) the same grip. A scenario that sets the weather on the tick loop sets the grip with it: a factor applied again replaces the last one rather than compounding it.

The grip is CARLA's vehicle physics alone: no traffic backend reads it.

The example scenarios declare these, each a node of their scenario_rules:

Scenario Measure Node Parameter
cut_in/* vehicle_ahead_gap_m vehicle_ahead_gap, 2-30 m npc_gap_m (target: measure)
cut_in/* vehicle_ahead_relative_speed_kph vehicle_ahead_relative_speed, -15 to 15 km/h npc_initial_speed_kmh, offset by the ego's
pedestrian_dart_out crossing_pedestrian_gap_m crossing_pedestrian_gap, 8-30 m trigger_distance_m
pedestrian_dart_out crossing_pedestrian_speed_ms crossing_pedestrian_speed, 0.8-3 m/s walk_speed_ms

traffic_light_compliance declares none, but has what a control needs: stop_line_distance_m (target: measure: the ego is moved along the lane its spawn pose is on until stop_line_distance_m reads it, place_at_stop_line_distance), and its own measure red_light_duration_s, the seconds every light is held red (light_switch_delay_seconds, a raw setting). With the delay drawn, timeout_after_green_seconds counts the fail-safe timeout from the green light instead of from the start. A drawn distance does not decide the verdict by itself: the run is judged by where the ego stops against that same line -- not past it on red, standing in the stop zone (stop_zone_min_m-stop_zone_max_m before it) by the green -- so an ego started far back may roll up to the line on red.

sweep.odd_sample.knobs takes the same fields keyed by the ODD's attributes, and replaces anything above, attribute by attribute, for that sweep. It is for the command line -- one sweep against one ODD -- not for a scenario's config, which would then name the ODD's taxonomy: a scenario says controls. Controls set where a run starts; the measures read what it then drives, which is what coverage counts.

Visibility follows the conditions

CARLA draws the weather but its traffic never looks through it: the TrafficManager's vehicles know where every actor is, fog or not. A driver does not, and drives so as to stop within the road they see. So how far drivers see is a quantity of the run, in metres, which the conditions bound (the weather rules' visibility). Each of the fog, the rain and the light gives a range, and the shortest holds:

Condition Visibility, m
clear day 250-400
fog visibility 15 / 50 / 150 / 400 m and over 12-18 / 40-60 / 120-180 / 250-400
rainfall rate 2.5 / 7.6 / 25 mm/h 180-320 / 100-220 / 60-150
dusk, dawn 150-275
night 50-150 (a dipped beam to a main beam or a lit street)

Between the rows the bounds are interpolated. It is worked out before the run, from the conditions the run names, at environment.visibility_quantile of its distribution (0 its low end, 1 its high end; the median when not set), and the run records it as environment.visibility_m, which also fixes it whatever the conditions. A sweep.odd_sample draws the quantile for every case, as it draws the grip -- a parameter of the case, quantile(visibility_quantile), which the flaky search moves -- so the ODD's rainfall rate, fog visibility and time of day are what shorten it: a night case's drivers see 50-150 m, a thick-fog case's 12-18 m. visibility: false in the odd_sample leaves every case at the median (read as tire_friction is, strictly). An ODD that names the visibility itself (the visibility probe) sets it through its own knob, environment.visibility_m, and a case with one keeps it, as does a run whose own overrides name the visibility or its quantile. A run that names no condition records none, and its traffic sees as it always did; each run starts with none, whatever the last left.

What follows it is the traffic backend's (traffic.vehicles.follow_visibility): the sumo backend holds its own traffic to the speed its drivers keep in it -- the weather rules' sight_speed, which reads the visibility -- by default (Visibility); the traffic_manager backend holds every vehicle it drives to it with follow_visibility: true, under its road's speed limit (Weather and the TrafficManager), and otherwise only logs the speed it would hold traffic to.

Weather rules

What the conditions do is a Bayesian network of conditional distributions, not code: weather_rules.yaml in the package (autoware_carla_scenario.weather_rules), the CARLA backend's own rules (CarlaBackend.rules), which openodd reads and evaluates in its Rust core (its rules docs have the file format in full). It is a version 3 file:

  • its causes are its inputs, concepts of the taxonomy it names (taxonomy: {bundled: default}, openodd's bundled subset of tier4/taxonomy): rainfall_rate (mm/h), fog_visibility_range (m), time_of_day (a label), wind_speed (km/h) and cloudiness (a label) -- weather_rules.WEATHER_INPUTS, which the environment keys rainfall_rate_mm_h, fog_visibility_range_m, time_of_day, wind_speed_kmh and cloud_cover give (weather_rules.CAUSE_SETTINGS);
  • its effects are its quantities, in no taxonomy: CARLA's weather fields (carla_precipitation, carla_cloudiness, carla_precipitation_deposits, carla_wetness, carla_wind_intensity, carla_fog_density, carla_sun_altitude, carla_sun_azimuth: weather_rules.CARLA_QUANTITIES), the grip (tire_friction), the visibility and the speed a driver keeps (sight_speed);
  • every part says what it rests on (basis): the default rules are an expert's, set from CARLA's presets and published ranges, not fitted.

A quantity is one or more parts; each reads some causes or other quantities and gives the quantity's distribution at points of them:

quantities:
  carla_precipitation:
    parts:
      rain:
        at: {rainfall_rate: [0, 0.1, 2.5, 7.6, 25]}
        uniform:
          low:  [0, 1, 25, 60, 95]
          high: [0, 10, 35, 75, 100]
  tire_friction:
    combine: min
    parts:
      wetness:
        at: {carla_wetness: [0, 50, 100]}        # dry, wet, flooded
        uniform:
          low:  [0.90, 0.65, 0.45]
          high: [1.10, 0.85, 0.65]
      standing_water:
        at: {carla_precipitation_deposits: [0, 50, 100]}
        uniform: {low: [0.90, 0.65, 0.45], high: [1.10, 0.85, 0.65]}
  • A part's distribution is uniform (low, high), triangular (low, mode, high) or normal (mean, sd, cut to low..high). Each parameter is one number or one value per point; between number points it is interpolated linearly, beyond them it holds the outermost point's value. A categorical cause (the time of day) is written at its labels, with nothing between them. So the default rules give exactly the tables above.
  • A part may read several inputs, with a value at every combination of their points, as nested lists.
  • A run's value is the one at its quantile: the value with that share of the distribution below it (weather_rules.weather_at). With more than one part, every part gives its value at the same quantile and combine (min, the default, or max) joins them: the visibility's fog, rain and light are three parts, and the shortest holds. The parts move together -- a case at quantile 0.1 is at the worse end of the fog's, the rain's and the light's alike -- so the quantity's range is the least of the parts' lows to the least of their highs.
  • A cause the run does not name reads its default: no rain, no fog (10 km), the day. weather_rules.resolve_environment works out only the fields that follow from what the run names, so a run that names the time of day alone keeps the map's rain.

Chains. A part may read another quantity, so the rules chain: the rainfall rate gives CARLA's precipitation, which gives its wetness and standing water, which give the grip, and its cloudiness -- the cloudier of the sky the run names and what the rain brings (combine: max); and sight_speed, the speed (m/s) a driver keeps, reads the visibility. Its distribution is triangular at each visibility: at most the speed SUMO's default car (braking at 4.5 m/s² a second after it sees) stops from within it, v * tau + v² / (2 * decel) = the visibility, and down to three quarters of that (visibility.sight_speed).

Visibility, m 15 50 100 250 400
sight_speed, km/h (low / mode / high) 22 / 27 / 29 46 / 59 / 62 70 / 88 / 93 117 / 148 / 155 150 / 190 / 200

visibility.sight_speed_at(visibility_m, quantile, decel, reaction_s) reads it at a quantile for a car that brakes and reacts otherwise: the rules' value scaled by that car's stopping speed over the default car's, and never above that car's stopping speed. A traffic backend holds each of its vehicles to it at a quantile of the vehicle's own (visibility.driver_quantile, from its id: the same in every run, spread over the traffic), so some drivers keep close to what they can stop from and others well below it. weather_rules.default_rules().sample(conditions) gives every quantity of the conditions at once, each at its median.

A rule file is checked when it is read: every field known, every input's points increasing, one value per point, the distribution valid at every point (low not above high, a mode between them, a positive sd), and every input a part reads a concept of the taxonomy (openodd.RulesError says where). A version 2 file still reads merged over the default rules (it keeps their version and taxonomy), but one that reads an input standing for no concept is refused.

A run's rules. environment.rules names a rule file -- a path, absolute or relative to the directory the run starts in, or an https:// URL -- merged over the default rules: it names only what it changes (a part's uniform without its at, say), and may extends other files, by path, URL (pinned with {url, sha256}) or git source. A sweep sets it like any other key (environment.rules=/data/grip_measured.yaml); every run sets the rules it names, or the default ones, when it starts, so one run's do not reach the next -- in a batch too, which builds every case before any runs: each case runs, and records in its coverage file, its own rules.

Rules by vehicle. Tyres differ by vehicle; the weather does not:

environment:
  rules:
    default: weather/measured.yaml          # every vehicle no entry names
    vehicles:                               # the first entry a vehicle matches
      ego: weather/ego_measured.yaml        # the ego (role_name "Ego")
      "vehicle.mini.*": weather/mini.yaml   # a blueprint pattern

Each vehicle's file is read over the run's default rules. default and every vehicles entry name one file each -- a path or an https:// URL, as environment.rules itself does; a pinned {url, sha256} is for extends alone, so a file to pin goes in one that extends it. Anything else (a number, a mapping, a list, an unknown key) is refused before the batch runs (openodd.RulesError, naming the key; weather_rules.check_weather_rules). Every EnvironmentAction (the config's environment, even one naming only rules, registers one) gives each vehicle the grip of its own rules, all at the same tire_friction_quantile (a case draws one; the median when none is set), and a vehicle spawned later its own too; the ODD's tire_friction probe reads the ego's. tire_friction, the grip named outright, is every vehicle's. Everything else -- CARLA's weather, the visibility -- is the default rules'.

weather_rules.weather_rule_files() lists every rule file a run follows, with the SHA-256 of each.

Traffic parameters from the ODD

How the traffic behaves -- how fast drivers go against the limit, how they react, how often they run a red light, how fast pedestrians walk -- is the run's traffic parameters (Traffic parameters). Each numeric one is a spread, a mean and a standard deviation, that every vehicle or pedestrian draws its own value from (a normal distribution cut to the parameter's bounds). An ODD bounds the spread: the mean and the std of each are built-in probes, with knobs, so an ODD that names them makes a sweep draw both per case, and every road user of the case then draws its own value from them.

Probe Knob (written) Knob range
vehicle_<field>_mean traffic.vehicles.<field>.mean the field's mean range below
vehicle_<field>_std traffic.vehicles.<field>.std the field's std range below
pedestrian_<field>_mean traffic.pedestrians.<field>.mean likewise
pedestrian_<field>_std traffic.pedestrians.<field>.std likewise
bicycle_<field>_mean traffic.bicycles.<field>.mean likewise (sumo only)
bicycle_<field>_std traffic.bicycles.<field>.std likewise (sumo only)
Parameter Unit Mean range Std range
vehicles.speed_factor 0.5-1.5 0-0.3
vehicles.min_gap_m m 0.5-10 0-3
vehicles.ignore_lights, ignore_signs, ignore_vehicles, ignore_walkers, random_lane_changes 0-1 0-0.5
vehicles.accel m/s^2 0.5-5 0-1.5
vehicles.decel m/s^2 1-9 0-2
vehicles.reaction_time_s s 0.3-3 0-1
vehicles.imperfection 0-1 0-0.5
vehicles.max_lateral_speed_mps m/s 0.2-2 0-0.5
vehicles.max_lateral_accel m/s^2 0.2-3 0-0.5
pedestrians.walking_speed_mps m/s 0.5-2.5 0-0.5
pedestrians.min_gap_m m 0.1-2 0-0.5
pedestrians.red_light_violation_s s 0-60 0-30
bicycles.max_speed_mps m/s 2-8 0-1.5
bicycles.accel m/s^2 0.5-2 0-0.5
bicycles.decel m/s^2 1-4 0-1
bicycles.min_gap_m m 0.2-2 0-0.5
bicycles.reaction_time_s s 0.3-2 0-0.5
bicycles.imperfection 0-1 0-0.3
bicycles.max_lateral_speed_mps m/s 0.2-2 0-0.5
bicycles.red_light_violation_s s 0-30 0-10

The bicycles' are traffic.bicycles (BicycleParameters, Traffic parameters): only SUMO simulates bicycles, so an ODD that draws them is an ODD for the sumo backend.

A bucket of the ODD's attribute narrows the range, as for any knob. In a binding file:

# binding.yaml
behaviour.speed_factor: {probe: vehicle_speed_factor_mean}
behaviour.speed_spread: {probe: vehicle_speed_factor_std}
behaviour.reaction_time: {probe: vehicle_reaction_time_s_mean}
behaviour.walking_speed: {probe: pedestrian_walking_speed_mps_mean}

A case drawn with behaviour.speed_factor in [0.8, 0.9) and behaviour.speed_spread in [0.05, 0.1) writes traffic.vehicles.speed_factor.mean and .std, and each vehicle of its run draws its own speed factor from that normal distribution. The probes read what the run set, every tick, so the run's coverage records the mean and std it drove with; a run that set none reads a missing value.

A parameter the run's backend does not support is still refused: an ODD that draws a reaction time is an ODD for the sumo backend, and a case of it run on the traffic_manager backend fails before it starts (Traffic parameters).

The road a case is on

A sweep with cases of its own -- constraints, or a route search -- draws the attributes the Lanelet2 map decides as well: whether the case is in a junction, how many lanes, the speed limit, the location and road type tags. Each is read off every case's lanelet (its first *lanelet_id override: the constraints' pick, or the route's spawn) with the probe's lanelet form, and gets a knob that reaches the buckets those lanelets are in. The knobs write no setting a run reads (_odd_map.*, taken off again); instead each drawn case is put on a case whose lanelet is in every bucket drawn, the one run least so far among those that are. So a lanelet case may run more than once, or not at all, even without a count: the buckets decide, not the list of cases. Each run records its case (++odd_case) with its own lanelet's buckets and values, not the ones drawn.

Not every combination of buckets is on one lanelet -- no junction where the limit is 30, say -- and each knob draws its attribute on its own, so 20 cases are drawn for each one kept and those no lanelet fits are passed over. If too few fit, the rest go on the lanelet in most of their buckets, with a warning. So strategy: coverage aims at the road as at everything else: the buckets of junction, lane count or speed limit that earlier runs left uncovered come first, as far as the sweep's cases reach them.

A sweep that gives an attribute a knob itself (odd_sample.knobs) keeps it. The flaky search (odd_sample.flaky) draws its cases as it chooses them, none passed over; each goes on the lanelet in most of its buckets.

How a case is drawn

  1. Each knob's attribute gets one of its buckets, leaving out the buckets outside the ODD (the ones scenario-odd show lists as out).
  2. The combination is kept only if the ODD's verdict on those buckets is not outside -- so modules that rule out a combination (no heavy rain at night, say) are honoured, not just single buckets.
  3. A value is drawn inside each bucket (a numeric bucket's interval, or the knob's range) and checked against the ODD again, as a value, with the attributes an OpenODD ODD derives from others worked out: a condition finer than a bucket (between(0, 45) across a [30, 60) bucket) holds for what is drawn. What is written is the value exactly as drawn, and a numeric one is read back to check it is still in its bucket.

strategy: uniform draws every admitted bucket alike. strategy: coverage reads coverage_from and draws each attribute from its least covered buckets first, counting the cases already drawn in the batch -- by earlier scenarios on the same ODD too, in a glob batch -- as covered, so a batch works through the holes before it repeats one; it falls back to any bucket (the less covered the likelier) only when the ODD admits no combination of the least covered. Situations that are still holes (or that no earlier run measured) are aimed at first, one case each; inactive ones are not, nor are those an earlier case of the batch was aimed at. A situation the knobs alone decide -- every attribute it tests has a knob, and it refers to no other module -- holds in its case. One that also tests what no knob sets -- a speed, a road -- gets a case under which it can hold; whether it does is up to the drive. A situation the ODD and the knobs leave no case for is given up on, with a warning.

coverage_from paths are relative to the directory the sweep is started from. Coverage is read when the sweep is expanded: to resume a sweep (--resume-from) into the same cases, read the same coverage -- point coverage_from at the earlier runs, not at the sweep's own output.

From Python:

from pathlib import Path

from autoware_carla_scenario.coverage.report import load_and_merge
from autoware_carla_scenario.odd import OddKnob, OddSampler, resolve_odd

sampler = OddSampler(
    resolve_odd("path/to/urban.yaml"),
    controls={"vehicle_ahead_gap_m": OddKnob("scenario.npc_gap_m", range=(2.0, 30.0))},
    seed=0,
    strategy="coverage",
    coverage=load_and_merge([Path("outputs")]),
)
for case in sampler.sample(20):
    print(case.overrides, case.buckets, case.situation)

To go on from cases drawn before -- another scenario's, on the same ODD -- pass them as drawn; OddDraws keeps them per ODD, as a batch does:

from autoware_carla_scenario.odd import OddDraws

draws = OddDraws()
draws.add(sampler.odd, sampler.sample(20))
later = OddSampler(sampler.odd, seed=0, strategy="coverage", drawn=draws.of(sampler.odd))

Sampling by shares of the ODD

weights gives chosen parts of the ODD a share of the cases, and draws the rest as strategy: uniform would (it implies strategy: weighted). The parts are the ODD's modules, named in its OpenODD files, of one condition or of several -- so the sweep names no attribute of the taxonomy, and a module redefined in the ODD needs no change to the sweep:

# regions.yaml: parts of the ODD, in plain OpenODD
IMPORT: [urban.yaml]
MODULES:
  heavy_rain:
    INCLUDE_AND:
      environment.rain: heavy
  rainy_night_merge:
    INCLUDE_AND:
      environment.rain: heavy
      environment.illumination: night
      scenery.road_type: merge
# shares/rainy.yaml
heavy_rain: 0.3            # 30% of the cases in heavy rain
rainy_night_merge: 0.1     # 10% in heavy rain at night in a merge
uv run scenario --multirun 'scenario=cut_in/*' odd=path/to/urban.binding.yaml \
  +sweep.odd_sample.count=50 +sweep.odd_sample.weights=shares/rainy.yaml
# or inline, for one module
  '+sweep.odd_sample.weights={heavy_rain: 0.5}'

The binding file names the regions file with the ODD's (openodd: [urban.yaml, regions.yaml]), and the ODD names its roots under ODD:: a module that is a root -- a document with no ODD: section takes every module that is not a situation as one -- would bound the ODD, and is refused a share (every case is in it). A relative path to a file of shares is resolved against the working directory the command is started from (a --multirun, a batch, scenario-expand), not the directory of the scenario's config, as coverage_from is: the path is given on the command line, or in a config the command line composes. Give an absolute path for a file next to a scenario package. A file that cannot be read or parsed, or that names a module the ODD does not have, refuses the sweep before any case runs, with one line Error: <scenario>: <why> and exit 1.

Each module holds its share of the cases, however the modules overlap: with heavy_rain: 0.3 and night: 0.3, 30% of the cases are in heavy rain and 30% at night, those in both counting for both. A share is a fraction in [0, 1], not a weight relative to the others. Inside and outside the modules the cases are drawn as the uniform strategy draws them (of all the ways to hold the shares, the one closest to it), rounded to whole cases counting the cases drawn before -- by the earlier scenarios on the ODD in a glob batch too, so ten scenarios drawing 1 case each with heavy_rain: 0.3 draw 3 in heavy rain between them.

Refused: a name that is no module, a root, a share outside [0, 1], weights with another strategy; a module whose truth the knobs do not decide (it tests an attribute nothing draws, such as the ego's speed); shares that cannot hold together inside the ODD (a: 0.7 and b: 0.6 for modules that never overlap). With flaky, the part of each batch the search does not explore holds the shares. A sweep with constraints whose map decides some attributes keeps, of more cases drawn, those on a lanelet of the sweep, so its cases hold the shares only roughly. The shares are the sampler's, not OpenODD's, which has no construct for a distribution (openodd's weighted sampling).

Finding flaky cases

Every run a sweep draws from the ODD records its case in its coverage file: the values it ran with, why it was drawn, and whether it passed (openodd's CoverageCollector.record_case and record_outcome; the runner calls them, so a scenario needs nothing of its own). The merged report then lists the runs, flakiest first, with their flakiness assessed over all the runs of their scenario (scenario-coverage, section Flaky cases).

With flaky, the sweep also looks for them. openodd's FlakySearch learns where the runs in coverage_from passed and failed (Gaussian-process classification), and draws part of each batch where a small change would flip the outcome -- after the sampler has covered every bucket, the rest of the batch still sweeping the ODD (openodd's docs/flaky.md):

sweep:
  odd_sample:
    count: 20
    strategy: coverage
    coverage_from: [outputs]       # the earlier sweeps' runs: what the search learns from
    flaky:
      explore_fraction: 0.8        # of each batch, once every bucket is covered
      scenario: CutInScenario      # learn from this scenario's runs only (all, when left out)

flaky: true takes the defaults; false, null or leaving it out is no flaky search. A switch is read as tire_friction's is, so "true", 1, "false" and 0 mean what they say, and anything else that is not a mapping is refused. The mapping's other fields are FlakySearch's: pool, coverage_depth, threshold, epsilon, perturbations, draws and directions. The space holds every parameter of a case: the attributes drawn, the weather effects' quantiles, and where in its range the grip and the visibility are (quantile(tire_friction_quantile), quantile(visibility_quantile)), so two cases alike but for their grip are two points, not one point whose outcome looks unsure. Only runs drawn in the same space -- the same attributes, ranges and labels -- with an outcome are learnt from (runs drawn before the grip and visibility were parameters, in a space without them, are not); a sweep with no such run draws all its cases with the sampler's own policy. Each case chosen for its instability records the flakiness predicted for it, beside the one the report assesses. Run the sweeps one after another, each reading the earlier ones' outputs, and every sweep looks where the last left the outcome unsure.

Checking an ODD

uv run scenario-odd check my_package.odds:urban_odd   # compile with Codon
uv run scenario-odd check path/to/urban.yaml          # read the binding and OpenODD files
uv run scenario-odd show path/to/urban.yaml           # attributes, buckets in and out
uv run scenario-odd list                              # ODDs known by name

check compiles a Python ODD with Codon, the same way a scenario is compiled. It catches the following:

  • a misspelt name;
  • a condition of the wrong type (between("0", 60));
  • a probe that does not take a world.

The runner makes the same check before a run, under the typecheck config key. OpenODD YAML is checked as it is read (see What is read).

Checking a planned route

Before a scenario runs, its ego route can be checked against the ODD on the Lanelet2 map alone, with no CARLA server:

uv run scenario-odd route path/to/urban.yaml intersection_passing/left_turn
uv run scenario-odd route urban 'intersection_passing/*' lane_change/left map=nishishinjuku
uv run scenario-odd route urban 'lane_change/*' --json

Each scenario config is composed as a run composes it. Name it as scenario= does (a glob works too); arguments with an = are Hydra overrides for all of them. The route is planned from the ego's spawn pose (ego.spawn_lanelet_id, ego.spawn_s), through ego.waypoint_lanelet_ids, to the goal (ego.goal_lanelet_id, ego.goal_s), as the shortest path in the map's routing graph, leg by leg. A scenario that derives its goal in setup() from scenario.expected_route_lanelet_ids, as intersection_passing does, gets that goal here too (a configured goal wins). A goal derived any other way is only known once the scenario runs: that route is reported as not planned. So is an exported scenario on a map of its own composed on another map (map=<id>_map left out), before any map is read: its lanelet ids are its own map's (Running an authored scenario).

The ODD's attributes are then read lanelet by lanelet along the route, and each lanelet gets one of three verdicts:

Verdict Meaning
inside The ODD holds there, whatever happens at run time.
outside The ODD fails there, whatever happens at run time. Reported with the lanelet ids and the modules that fail.
undecided It depends on what only the run can tell (weather, traffic, speed), or on a value the map does not have. At run time a missing value counts as inside, so this is "inside only by assumption".

The map decides an attribute when its probe has a lanelet form, probe.on_lanelet(lanelet, lanelet_map, routing_graph). It returns the value, None where the run would read none, or UNDECIDED where only the run can tell (for example speed_limit_kph on a lanelet with no speed_limit tag, where the run falls back to CARLA's limit). The built-in probes listed above have one; a custom probe opts in by having an on_lanelet attribute (a function attribute, or a method of a callable object). An attribute whose probe has none is open: it could take any value, and the ODD's own three-valued evaluation decides, exactly as it does for buckets outside the ODD. An OpenODD attribute derived from others is worked out from their values on the lanelet, and is open while one of them is.

What the lanelet forms read, and where that differs from the run:

Probe On a lanelet At run time
in_junction The lanelet has a turn_direction tag (Autoware tags every lanelet in an intersection with one): lanelet_junction.is_junction_lanelet, which the sweeper's is_junction, its bindings and the route search ask too, so where a sweep places a scenario and what a run measures never disagree The same, on the ego's lanelet; without a Lanelet2 map, CARLA waypoint is_junction
junction_type intersection where in_junction is true, else nothing The same, by in_junction
lane_count The lanelet and every same-direction neighbour, walked left and right in the routing graph (left/right, else adjacentLeft/adjacentRight, so lanes behind a solid line count too); nothing in a junction The same, on the ego's lanelet; without a Lanelet2 map, the driving lanes of the same sign on the CARLA waypoint's OpenDRIVE road
speed_limit_kph The speed_limit tag, else (no tag, or not a finite number above 0: nan, inf, 0, -30) undecided The tag, else (the same) CARLA's get_speed_limit()
lanelet_location, lanelet_subtype, lanelet_speed_limit_kph The tag The tag of the lanelet the ego is on

A run with a Lanelet2 map reads the junction and the lane count off the ego's lanelet, as the route check does, so the two agree. A run without one falls back to CARLA's OpenDRIVE waypoints. The two maps are converted from each other but not always lane for lane (two lane-change neighbours on the Lanelet2 map can be the single lanes of two OpenDRIVE roads), so there the route check and the run can disagree.

The report also gives the expected coverage: the metres the route spends in each bucket of each attribute (<undecided> and <missing> collect the metres where the map cannot or does not say). The start lanelet counts from the spawn pose, the goal lanelet up to the goal pose, and a lane change splits the lanelets' length between the two sides. Over several scenarios the metres are summed, and the report lists the buckets of each map-decided attribute that no planned route reaches, leaving out buckets outside the ODD (they are not targets). An attribute the map decides in general but left undecided on some lanelet of a route (metres in <undecided>) is reported as undetermined instead: the run may reach any of its buckets there, so none is called unreached.

The exit status is 0 when no route leaves the ODD, 1 when one does, and 2 when a route could not be planned (no goal, no path, a lanelet the map does not have, no Lanelet2 map) or the ODD could not be read.

From Python:

from autoware_carla_scenario.odd import (
    PlannedRoute, combine_route_coverage, plan_route_coverage,
)
from autoware_carla_scenario.coordinate import Lanelet2Pose

route = PlannedRoute(start=Lanelet2Pose(203, 25.0), goal=Lanelet2Pose(207, 10.0))
coverage = plan_route_coverage("urban", route, lanelet_map=lanelet_map)
coverage.leaves_odd, coverage.outside(), coverage.expected_m
combine_route_coverage("urban", [coverage, ...]).unreached

plan_route_coverage also takes a scenario (its spawn pose, waypoints and goal) or a list of lanelet ids. Without lanelet_map it uses the map the runner has loaded.

At start-up, once setup() has settled the goal, the runner plans the route the same way and logs a warning when it leaves the ODD. It is a warning only. It is skipped for an ODD with no modules, and anything that keeps the route from being planned is logged at debug level; the run goes on regardless.

What the report says

For each ODD the merged runs were measured against, scenario-coverage says the following (Coverage):

  • how long the runs were inside it, inside only because values were missing, and outside it;
  • which modules ruled ticks out, and how often, and how often their verdict rested on missing values;
  • which runs left it, and when (the first intervals);
  • which attributes are monitored but not covered.

A bucket is outside the ODD when the ODD fails for every value in it, whatever the other attributes are. It is reported, but it is not a coverage target. An example is nonurban under location.is_in(["urban"]).

To decide this, the ODD's own evaluation runs with the bucket in place of the attribute's value and every other attribute left open. Labels, inactive modules, groups and module references therefore count exactly as they do on a tick. A condition that ties attributes together (any_of across two of them, or a junction and a speed) rules out combinations, never a single bucket. A bucket only partly inside ([60, 90] under <= 60) stays a target.