ODD¶
An Operational Design Domain (ODD) states the conditions a system is built to drive in: urban roads, up to 60 km/h, no heavy rain, and so on. Every run is measured against one. The runner samples the ODD's attributes on every tick. Their buckets are the run's ODD coverage: which conditions it reached (see Coverage). The ODD's modules say which ticks were outside the ODD.
An ODD is written in one of two ways. Both build the same objects, so they are measured, judged and reported the same way:
- Python: openodd's model (Writing an ODD in Python), with this framework's probes. It is checked with Codon before it is used, like a scenario (Static Check).
- ASAM OpenODD 1.0 YAML (the standard's chapter 10). It is read into the same objects. A binding file next to it names the probes that measure its concepts.
OpenODD YAML + binding file ──(load_odd_binding)──┐
├──▶ OddDefinition ──▶ coverage + inside/outside per tick
Python (checked by Codon) ─────────────────────────┘
OpenODD also maps its model to OpenSCENARIO DSL (chapter 9, model to DSL only) and to tables (chapter 8). Neither is read here.
The reading, judging, versioning, sampling and coverage are
openodd (openodd-py, a Rust
core with a Python API), which knows nothing of CARLA. This framework is its
backend: autoware_carla_scenario.odd.backend.CarlaBackend, a subclass
of openodd.Backend, gives the built-in probes a binding file names,
measure: keys read from the running scenario, the knobs that set the
weather and the sun, and where the ego is (for coverage). Importing autoware_carla_scenario.odd
makes it openodd's default backend, so openodd.load_odd_binding() and
autoware_carla_scenario.odd.load_odd_binding() read the same.
Versions of an ODD¶
OpenODD 1.0 defines no version of an ODD; openodd builds one on what it does define (its versioning profile):
- the revision is the SHA-256 of the files the ODD was read from -- the
binding file, the OpenODD files it names and every file they
IMPORT-- by name and content. The same files are the same revision on any machine and from any repository or commit; any change to any of them is a new one; - the version is what the root modules (those under
ODD:) declare in theirMETADATA(version: 2.1.0); - the structure is the digest of the ODD as built (modules, buckets, the buckets outside it), for an ODD written in Python, which has no files.
All three are written to every coverage file (the ODD's revision,
version, structure), so coverage can be told apart by the ODD it was
measured against. OddDefinition.archive() packs the files into a
deterministic .tar.gz that openodd.load_odd_archive() reads back exactly;
a run writes it beside its coverage file ({ScenarioName}_odd.tar.gz).
Concepts¶
The model and its semantics follow OpenODD 1.0 (chapters 6 and 7).
| Concept | What it is |
|---|---|
| Attribute | A taxonomy concept, measured on every tick: a probe reads it from the world, and its buckets make it a cover item. |
| Condition | A test on one attribute (speed_limit.between(0, 60)), a group of conditions (all_of, any_of), another module's verdict (module_holds), or a missing value (is_unknown). |
| Module | A named rule: INCLUDE AND (NOT EXCLUDE). It has at most one include section and at most one exclude section. |
| Label | A name several modules declare. It holds when any active module declaring it holds. |
| Root | An entry point. Each root candidate (roots=; by default every module) that no other module refers to, by name or through a label, is a root. The ODD holds when its active roots hold. |
Missing values. OpenODD's missing-value semantics apply. A value the
probe could not read does not, by itself, put a situation outside the ODD
(open world). Examples are a simulator without weather, and a run without a
Lanelet2 map. When a value is required, the ODD says so with
attr.is_unknown() in an exclude section (x: unknown in YAML). A tick that
is inside only because values were missing is reported as such.
Inactive modules. An inactive module is ignored. A condition referring to it, either way, drops out of its section, as if it were not written. A section with nothing left in it counts as absent: an include section then holds, and an exclude section does not. The standard says such conditions are "automatically satisfied"; taken literally, that would make an exclude section that refers to a switched-off hazard exclude everything, so this reads "ignored" as the neutral element.
Modules follow ISO 34503's "default" definition mode, as OpenODD requires. Whatever no module rules out is inside the ODD.
Writing an ODD in Python¶
An ODD is a function that takes nothing and returns an OddDefinition. The
model, the registry and the reading of a name are openodd's (its
Writing an ODD in Python);
this framework re-exports them, gives them its probes, and checks the ODD
with Codon:
from __future__ import annotations
import typesafe_carla.carla as carla
from autoware_carla_scenario import (
OddAttribute,
OddDefinition,
OddModule,
module_holds,
register_odd,
)
from autoware_carla_scenario.odd import (
INTENSITY_LEVELS,
lanelet_location,
rain,
speed_limit_kph,
)
def ego_yaw_rate(world: carla.World) -> float | None:
... # any function of the world is a probe; None means unknown
def urban_odd() -> OddDefinition:
location = OddAttribute(
"scenery.location", lanelet_location, values=["urban", "nonurban", "private"]
)
speed_limit = OddAttribute(
"scenery.speed_limit", speed_limit_kph, unit="km/h",
buckets=[0, 30, 40, 50, 60, 80, 100],
)
weather = OddAttribute("environment.rain", rain, values=INTENSITY_LEVELS)
yaw_rate = OddAttribute(
"dynamic.yaw_rate", ego_yaw_rate, unit="deg/s", range=(-30.0, 30.0), every=10.0
)
return OddDefinition(
"urban",
[location, speed_limit, weather, yaw_rate],
[
OddModule(
"urban_roads",
include_and=[location.is_in(["urban"]), speed_limit.between(0, 60)],
text="Urban roads up to 60 km/h",
),
OddModule("weather", exclude_or=[weather.is_in(["heavy"])], labels=["fair"]),
OddModule(
"root", include_and=[module_holds("urban_roads"), module_holds("fair")]
),
],
roots=["root"],
text="Urban roads, fair weather",
)
register_odd("urban", urban_odd)
Attributes¶
OddAttribute(name, probe, *, unit, range, every, buckets, values, text)
takes the bucket arguments of
register_cover(). Give at most one
of values, buckets and range (with every). An attribute with none of
them is monitored but not covered: conditions may test it, and the report
lists it so the blind spot is visible. Its cover item is named
"odd." + name.
Conditions¶
| Method | Holds when |
|---|---|
attr.is_in(values) |
the value is one of values (compared by label: an enum by name, a bool as true/false) |
attr.equals(value) |
the value equals value |
attr.between(low, high) |
low <= value <= high |
attr.at_least(x) / attr.at_most(x) |
value >= x / value <= x |
attr.greater_than(x) / attr.less_than(x) |
value > x / value < x |
attr.is_unknown() |
the value is missing |
all_of([...]) / any_of([...]) |
all of / any of the conditions hold |
module_holds(name, holds=True) |
the module (or label) named name holds, or does not |
Numbers are in the attribute's unit.
Modules¶
OddModule(name, *, include_and, include_or, exclude_and, exclude_or, labels,
active, text, situation, target, cover_by, min_stay). Give at most one of
include_and / include_or, and at most one of exclude_and / exclude_or.
Situations¶
Attribute coverage counts each attribute on its own: three lanes and 60 km/h can both be covered by runs that never drove 60 km/h on a three-lane road. A module already says such a combination, so a module can be covered as a situation:
OddModule(
"three_lane_cruise",
include_and=[lanes.equals(3), speed.between(55, 65)],
situation=True,
cover_by="meters", target=200, min_stay=3.0, # 200 m, counting stays of 3 s or more
)
Every tick, the module's verdict (which the ODD works out anyway) is a sample
of the item odd.situation.three_lane_cruise, which has one bucket, holds.
A tick where the module holds is a sample in it. A tick where it fails, or is
inactive, ends the stay. A tick where it rests on missing values ends the stay
too, and is counted apart as unknown. target, cover_by and min_stay
work as for attributes (docs/coverage.md). The report lists situations in a
group of their own, Situation coverage.
A situation is what the runs should drive, not a bound of the ODD: with
roots left out, a situation nothing refers to is not a root candidate. To
make it a bound too, refer to it from a root, or name it in roots.
Built-in probes¶
These live in autoware_carla_scenario.odd:
| Probe | Returns | Read from |
|---|---|---|
ego_speed_kph |
float, km/h | Ego velocity |
speed_limit_kph |
float, km/h | Lanelet2 speed_limit tag, else CARLA |
lanelet_speed_limit_kph |
float, km/h | Lanelet2 speed_limit tag |
lanelet_location |
str | Lanelet2 location tag (urban, nonurban, ...) |
lanelet_subtype |
str | Lanelet2 subtype tag (road, highway, ...) |
road_type |
highway, or nothing |
The taxonomy's road type, from the Lanelet2 subtype: highway for a highway; Lanelet2 has no tag that tells a local road from a primary one, so those read nothing |
drivable_area_type |
DRIVABLE_AREA_TYPES |
The taxonomy's drivable area type, from the Lanelet2 subtype: road and highway a lane, road_shoulder a shoulder, parking a parking_space, play_street mixed_pedestrian_and_vehicular_traffic |
in_junction |
bool | Lanelet2 turn_direction tag of the ego's lanelet, else (no Lanelet2 map) the CARLA waypoint |
junction_type |
intersection, or nothing |
The taxonomy's junction type (the default ODD's): intersection where in_junction reads true; Lanelet2 tells no branch or merge, and outside a junction it reads nothing |
lane_count |
int | Driving lanes in the ego's direction, outside junctions: the ego's lanelet and its neighbours in the Lanelet2 routing graph, else (no Lanelet2 map) the CARLA waypoint's OpenDRIVE road |
rainfall_rate_mm_h |
float, mm/h | The rainfall rate the run was set to (environment.rainfall_rate_mm_h); nothing when it named none |
fog_visibility_range_m |
float, m | The visibility through the fog the run was set to (environment.fog_visibility_range_m); nothing when it named none |
time_of_day |
TIMES_OF_DAY |
The sun: day above the horizon, dawn (rising, azimuth 0-180) or dusk (setting, 180-360) down to 6 degrees below it, night under that |
wind_speed_kmh |
float, km/h | The wind speed the run was set to (environment.wind_speed_kmh); nothing when it named none |
cloud_cover |
CLOUD_COVERS |
The cloud cover the run was set to (environment.cloud_cover), whatever the rain adds to CARLA's cloudiness; when it named none, CARLA's cloudiness: clear below 20, overcast from 70, partly_cloudy between |
road_surface_condition |
ROAD_SURFACE_CONDITIONS |
CARLA's water: standing_water from 50 standing water, wet from 10 of either, else dry |
illumination |
ILLUMINATION_LEVELS |
Sun altitude |
rain, fog, wind, cloudiness |
INTENSITY_LEVELS |
CARLA weather (0-100): precipitation, fog_density, wind_intensity, cloudiness |
road_wetness |
INTENSITY_LEVELS |
CARLA's wetness (0-100) alone: the standing water the rain brings is the rain's |
tire_friction |
float, share of the dry road's grip (1.0) | The wheel friction the run's weather left the vehicles; nothing when nothing set it |
visibility |
float, m | How far drivers see, as the run's weather left it (Visibility follows the conditions); nothing when the run set none |
traffic_density |
TRAFFIC_DENSITY_LEVELS |
Other vehicles within NEARBY_RADIUS_M |
nearby_vehicle_count |
int | Other vehicles within NEARBY_RADIUS_M |
pedestrian_nearby |
bool | A walker within NEARBY_RADIUS_M |
vehicle_<field>_mean, vehicle_<field>_std, pedestrian_<field>_mean, pedestrian_<field>_std, bicycle_<field>_mean, bicycle_<field>_std |
float | The mean and std of a traffic parameter's spread, as the run set it (Traffic parameters from the ODD); nothing when the run set none |
The Lanelet2 probes need the run to have a Lanelet2 map (map.lanelet2_path).
Without one they return nothing, except in_junction, junction_type and
lane_count, which then read the CARLA waypoint, and speed_limit_kph,
which then reads CARLA's get_speed_limit() for the ego.
The probes whose value the map alone decides (speed_limit_kph,
lanelet_speed_limit_kph, lanelet_location, lanelet_subtype,
road_type, drivable_area_type, in_junction, junction_type,
lane_count) can also
read it off a lanelet, which is how a
planned route is checked before a run
(Checking a planned route) and how a sweep picks
the road a drawn case is on (The road a case is on).
What they read off a lanelet is public too, for code that has to judge a
lanelet exactly as the ODD does: lanelet_in_junction(lanelet) (a
turn_direction tag, the default ODD's junction_type), lanelet_road_type(lanelet) (the
subtype tag, which the road type and the drivable area type follow) and lanelet_speed_limit(lanelet) (the
speed_limit tag, km/h; None for a tag that is not a finite number above 0,
as for a missing one). The adversities' location gate reads them
(autoware_carla_scenario.adversity.location).
Import them from autoware_carla_scenario.odd itself
(from autoware_carla_scenario.odd import rain). The Codon model that checks
an ODD has no probes submodule.
Scenario measures¶
The road users a scenario stages -- a vehicle ahead in the next lane, a
pedestrian running out -- are not read by ODD probes: they are measures
of the scenario, under keys the framework fixes
(autoware_carla_scenario.measures). Every BaseScenario has these, read
generically around the ego:
| Key | Unit | Version | Reads |
|---|---|---|---|
vehicle_ahead_gap_m |
m | 2 | The free road to the nearest vehicle ahead on the ego's lane or a lane beside it: its rear bumper less the ego's front bumper, along the ego's lane on the Lanelet2 map (up to AHEAD_RANGE_M); a vehicle about to cut in is measured from before it changes lanes |
vehicle_ahead_relative_speed_kph |
km/h | 2 | That vehicle's speed along the ego's lane less the ego's |
crossing_pedestrian_gap_m |
m | 1 | How far ahead the nearest pedestrian ahead that is moving was when it set off, held while it crosses |
crossing_pedestrian_speed_ms |
m/s | 1 | That pedestrian's speed |
stop_line_distance_m |
m | 1 | Along the ego's lane, from its front bumper to the nearest stop line ahead -- a traffic light's, a traffic sign's, a detection area's or a road marking's, as a stop_line_offset binding finds them -- up to AHEAD_RANGE_M, on any branch of the lane where it forks (the measure does not know the route); nothing without a lane to read along |
The pedestrian measures read in the ego's frame: ahead along its heading,
within PEDESTRIAN_LATERAL_M of its centre line.
The vehicle ahead¶
The vehicle-ahead measures read along the road, as a driver would say "the car in front is 15 m ahead", and from the map:
- The ego's lane is the lanelet the ego is on -- of those under its
origin (within
ON_LANELET_M), the one whose centreline runs closest to its heading, withinEGO_LANE_HEADING_DEG-- with every lanelet following it up toAHEAD_RANGE_Mahead and the lanelets behind it. Where the lane branches, every branch is the ego's lane: the measure does not know the route, so a vehicle ahead on either branch counts. Behind, the lane runs back one way only, through the first of the lanelets before each (Lanelet2'sprevious(...)[0]), where lanes merge; a lanelet joining the ego's lane is not its lane, nor beside it. Its arc lengthsruns along their centrelines, on from one to the next. - A vehicle counts when its origin is on one of those lanelets, or on a lanelet beside one of them -- its left or right neighbour in the Lanelet2 routing graph, whether a lane change is allowed across the line or not -- and further along the ego's lane than the ego's origin. A lane two over, an oncoming lane or a vehicle off the lanelets does not count, however near.
- The gap is
sof the vehicle's rear bumper lesssof the ego's front bumper. A bumper is the middle of the end of the actor's bounding box, along its own heading (Bumpers,bumpers_of), and it is projected onto the ego's lane: onto the lanelet the vehicle is on, or the one it is beside, and those before and after it. So the gap does not depend on where the lanelets beside each other are cut or how long they are -- the outer lane of a bend is longer than the inner one, and the sameson the two is not side by side -- and on a bend it is the distance along the road, not a straight line. A vehicle on the ego's own lane that overlaps it reads a negative gap; one in a lane beside that overlaps it (its rear bumper behind the ego's front bumper) is alongside, not ahead, and does not count, so it never hides the vehicle that is ahead. One beside and ahead of the ego counts, which is how a vehicle about to cut in is measured before it changes lanes. - The relative speed is the vehicle's velocity along the lane's tangent at its rear bumper less the ego's along the tangent at its front bumper: two vehicles keeping pace round a bend read 0 though their headings differ.
Without a lane to measure along -- no Lanelet2 map loaded, or the ego on no
lanelet that runs its way -- the measures fall back to the geometric
definition: a vehicle ahead of the ego's origin within three
LANE_HALF_WIDTH_M of its centre line counts, and the gap is bumper to bumper
along the ego's heading. The run logs a warning the first time, and the
coverage file says how many frames each definition read
(measures.vehicle_ahead_gap_m.definitions: along_lane, geometric;
vehicle_ahead_definitions() in Python). The geometric definition, too,
does not count a vehicle alongside in the next lane.
A reading costs about 3 ms with ten vehicles around the ego on CARLA
Town04_Opt (it was some 170 ms).
The ego's lane is found once and kept while its lanelet is still the ego's
-- the ego on it, running its way, and no lanelet overlapping it (found by
geometry, however the map cuts a fork's branches or an exit into lanelets)
running closer to the ego's heading -- and the whole
map is searched only when it is not (or, off the lanelets, once the ego has
moved 2 m or turned 5°); a vehicle is placed by
the lane's own lanelets, not by searching the map; and a projection reads
only the stretch of a centreline near the point. The Lanelet2 routing graph
the lane is read from -- some 25 s to build on Town04_Opt -- is built in the
run's setup whenever the ODD maps a vehicle-ahead measure that the
scenario reads with the built-in one -- not for one it replaced with
register_measure (prepare_measures) -- with every lanelet's bounding box,
not on the first tick. A failed reading reads nothing and is warned of once a run.
Measure versions¶
A measure's version (Measure.version, measure_version(key)) says
which definition it reads with, and is raised whenever what it reads changes
meaning. The vehicle-ahead measures are version 2. Version 1 (before
autoware-carla-scenario 12.0) read the gap centre to centre along the ego's
heading -- 4.75 m more than version 2 for a Lincoln MKZ behind a Mini on
CARLA 0.10 -- and counted a vehicle within three half-lanes of the ego's
centre line, whatever the map.
Coverage keeps the versions apart. openodd's merge treats two coverage items
as the same when their name, buckets, event and criteria agree; it does not
read their description. So the cover item of an ODD attribute mapped onto a
measure past version 1 is written under its version,
odd.dynamic.vehicle_ahead_gap@v2: merged with files written before, the
report lists odd.dynamic.vehicle_ahead_gap (version 1) and
odd.dynamic.vehicle_ahead_gap@v2 (version 2) apart, never one bucket's hits
added to the other's. The mark is @v<n>, not #<n>: #<n> is the suffix
openodd's merge gives an item it meets defined differently (version 1 files
whose buckets changed merge as odd.dynamic.vehicle_ahead_gap#2), and
openodd's sampler adds up an item's #<n> definitions. A coverage-driven
sweep (strategy: coverage) reads only the current version: OddSampler
hands openodd's sampler the @v2 items under the attribute's own item name
and leaves the older ones out (current_measure_coverage), so a
coverage_from that also holds version 1's files is safe. The 12.0.0
release wrote version 2 as #2; those files read as version 1, and their
hits are left out of a coverage-driven sweep on purpose: since 12.0.0 a
vehicle alongside in the next lane no longer counts as ahead, which changed
what version 2 reads, not only its name. Each
coverage file also lists the
measures its ODD reads under measures, with their version, unit and text.
A scenario that replaces a built-in measure (register_measure) promises the
same quantity, so its replacement carries the built-in's version.
Each reads nothing (missing) while there is no such road user. A scenario that knows better replaces one -- it knows which actor is its cut-in vehicle -- or adds one of its own:
class MyCutIn(BaseScenario):
def setup(self) -> None:
...
self.register_measure(VEHICLE_AHEAD_GAP_M, self._gap_to_my_npc)
An ODD maps its taxonomy onto measures: in Python with
scenario_measure(key) as the attribute's probe, in a binding file with
measure: in place of probe::
OddAttribute(
"dynamic.vehicle_ahead_gap",
scenario_measure(VEHICLE_AHEAD_GAP_M),
unit="m",
buckets=[0, 10, 20, 30, 50, 100],
)
The attribute then reads the running scenario's measure, its replacement
included, in the measure's unit: an attribute mapped onto a built-in measure
takes its unit, and one given that is not it is refused. The keys are in the
Codon model (from autoware_carla_scenario.measures import VEHICLE_AHEAD_GAP_M),
so a scenario or a Python ODD naming them is checked like any other
(Static Check): autoware_carla_scenario.measures is modelled
whole, Measure, BUILT_IN_MEASURES, read_measure(), check_measure_key()
and measured_scenario() included. A Measure is frozen, as a Metric is:
assigning to one of its fields fails the check, as it fails the run. A
scenario may build its own (Measure("my_measure", read, "m")); the check
calls its read with a world and refuses one that returns anything but a
float or an Optional[float]. A metric's keys are modelled the same way
(Metrics): autoware_carla_scenario.metrics is modelled whole, Metric and BUILT_IN_METRICS included. The ODD names a measure key, never a scenario; the scenario names
its measures, never the ODD. The default ODD maps none: what a scenario
sets up is no condition of the domain, but an effect the rules draw
(openodd's docs/rules.md, "Effects a scenario sets"), and a case records the
value it drew (case.quantities), which the logical parameter space is
plotted over.
Measures and metrics¶
A measure reads a condition the ego drives in: what the scenario sets up.
How the ego drives -- whether it collides, leaves the road, drives against
the traffic, how close it comes to other road users -- is an outcome, read by a
metric (Metrics). An ODD cannot map onto a metric:
scenario_measure("offroad"), or measure: offroad in a binding file, is
refused. Were it allowed, the ego's own driving would decide whether a run is
inside the domain it is tested in, and an ODD excluding offroad would report
a run that left the road as outside the ODD rather than as failed. A run
is judged on a metric with MetricCondition instead, once it is over.
Writing an ODD in OpenODD YAML¶
The OpenODD files are plain OpenODD 1.0. The example below follows the standard's own examples (Code 170, 171 and 192):
# taxonomy.yml
TAXONOMY:
environment_conditions:
rainfall_rate: float precipitation_rate
rainfall_level: # literals defined by ranges: ordered
no_rain:
rainfall_rate: "< 0.1 mm/h"
light_rain:
rainfall_rate: "[0.1 .. 2.5] mm/h"
heavy_rain:
rainfall_rate: "> 2.5 mm/h"
wind_speed: float velocity
scenery:
road_type: [road, highway, road_shoulder]
lane_count: integer count
# odd.yml
IMPORT:
- taxonomy.yml
ODD:
odd1:
TITLE: The baseline ODD
INCLUDE_AND:
low_speed_roads: true
needs_wind: true
EXCLUDE_OR:
bad_weather: true
MODULES:
low_speed_roads:
INCLUDE_AND:
road_type: [road]
lane_count: "< 3"
bad_weather_1:
LABEL: bad_weather
INCLUDE_OR:
rainfall_level: ">= heavy_rain"
wind_speed: "> 50 km/h"
needs_wind:
EXCLUDE_OR:
wind_speed: unknown # the wind speed is required
The binding file is this framework's own. It says which probe measures each concept, and its buckets:
# urban.yaml
openodd: [odd.yml] # relative to this file
name: urban # default: this file's stem
text: Urban roads, fair weather
probes:
road_type: {probe: lanelet_subtype}
rainfall_rate: {probe: my_package.probes:rain_rate_mm_h, unit: mm/h}
wind_speed: {probe: my_package.probes:wind_mps, unit: m/s}
lane_count: {probe: lane_count, values: [1, 2, 3, 4], cover_by: meters, target: 200, min_stay: 2}
Run with odd=path/to/urban.yaml. An OpenODD file on its own can be named
too, but then nothing is measured: every concept is missing.
Taxonomies and modules from git¶
An openodd entry can name a file in a git repository at a revision, so a
taxonomy or a module library kept elsewhere is used as it was at that
revision:
# urban.yaml
openodd:
- git: https://example.com/odd/taxonomy.git
rev: 'v1.2.0' # a tag, a branch, or a commit
path: taxonomy.yml # a list for several files
- git: git@example.com:odd/modules.git
rev: '3f2a9c1e0b7d4a6c8e5f1a2b3c4d5e6f7a8b9c0d'
path: odd/urban.yml
- local_overrides.yml # relative to this file
probes:
...
- An
IMPORTis looked for next to the importing file first, and then in the other sources: the directory of each file named, and the root of each repository. The standard makes file names unique within one transmission, soodd/urban.ymlin one repository canIMPORT: [taxonomy.yml]from another. A name found in two sources is refused. git fetchdoes the fetching, sourlis anything it takes, with the credentials it is configured with. Prompts are turned off, and theext::transport is refused.- Repositories and checkouts are cached in
$AUTOWARE_CARLA_SCENARIO_ODD_CACHEwhen it is set, else openodd's$OPENODD_CACHE(default~/.cache/openodd). A full commit id checked out once is read from the cache without the network; a tag or a branch is fetched on every load, to learn which commit it names now. Pin a commit for runs that must be reproducible. - Quote
rev: unquoted, a commit id of digits and onee(1630255e36) or a long one of digits only is read as a number, and looked for as one (1.630255e+36). - Fetching needs
git, which the scenario image does not have. A run there can still read a source pinned to a full commit id, from a cache filled beforehand (scenario-odd show urban.yamlwith the same cache directory) and mounted at$AUTOWARE_CARLA_SCENARIO_ODD_CACHE. - The commit each source was read at is written to the coverage file (the
ODD's
sources), andscenario-coveragelists it, warning when the merged runs read different commits of a source.
From Python, pass GitSource(url, rev, path) to load_openodd().
What is read¶
IMPORT: other files, relative to the importing one, or else in the
other sources (see above). A cycle is refused.
Every file read with an ODD makes up one transmission, in the standard's words, and the standard makes two things unique within one. Both are refused rather than guessed at:
- two different files of the same name, wherever they come from;
- a concept defined in two files, even the same way. A file can still add
concepts to a container another file started (
weather:in both, with different concepts under it).
TAXONOMY: nested mappings are records and containers. A leaf is one of:
integer|long|float|double <unit type>: a number;boolean;- a list of literals: a categorical;
- a mapping of literals to expressions on other concepts: a categorical
defined by them. Its value is the literal whose expressions hold; on a
shared range endpoint, the literal written first wins. Defined by ranges,
its literals are ordered in the order written, so
"< heavy_rain"means the literals before it; - the id of a categorical: the same literals.
A reference to a record (a user-defined type) and a shapefile are not
followed.
MODULES and ODD: modules. Those under ODD are the root
candidates: the ones among them no other module refers to are the roots. The
standard's examples also put referenced modules there. Without an ODD
section, every module no other module refers to is a root.
A module's keys are:
TITLE,DESCRIPTION;ACTIVE;LABEL/LABELS(one name or a list);METADATA(ignored, here and inside a section);- one
INCLUDE_ANDorINCLUDE_OR, and oneEXCLUDE_ANDorEXCLUDE_OR.
A section nests one level of the other operator: OR: in an AND section,
AND: in an OR one.
A section maps a concept to an expression, or a module or label to
true/false. A concept is named by its id (wind_speed), or by as much of
its path as makes it unique.
| Expression | Meaning |
|---|---|
low, 3, true, "0 mm/h" |
equal |
[a, b] |
one of the literals |
"> x unit", ">= x", "< x", "<= x" |
a bound (> excludes x, >= includes it) |
"[low .. high] unit", "[low, high] unit" |
an inclusive range |
"< literal", "[a .. b]" |
a bound or range over an ordered categorical |
unknown (none, null, undefined, or YAML null) |
the value is missing, unless the categorical has a literal of that name |
A number's unit must measure the concept's unit type. It is converted into
the probe's unit, so "> 50 km/h" against a probe in m/s is > 13.89. The
units known are listed below; a document adds more with OpenODD's
conversion: block.
| Unit type | Units |
|---|---|
| length | m, km, cm, mm, mi, in, ft |
| velocity | m/s, km/h (kph), mph |
| acceleration | m/s^2, g |
| time | s, ms (msec), min, h |
| angle | deg, rad |
| precipitation_rate | mm/h (mm/hr) |
| temperature | C, K, F |
| fraction | % |
| bandwidth | bps, kbps, Mbps, Gbps |
| frequency | Hz, kHz |
| illuminance | lx |
| count, occurrence | count, 1/h, occ/hr |
The following are refused, each with the module it appears in:
- a concept that is not in the taxonomy;
- a literal a categorical does not have;
- a unit of the wrong type;
- a second include or exclude section;
- deeper nesting;
- a module or label id containing
unknown.
The YAML is read with YAML 1.2 booleans: only true and false are
booleans, so literals such as off or NO stay literals. A key written
twice is refused. As the standard notes, an unquoted expression starting
with > (speed: > 50 km/h) is a YAML block scalar, so quote it.
Not read: COD / OD records, numeric terms (1.75*ego_width), $
parameters, and condition-level metadata.
The binding file¶
| Key | Meaning |
|---|---|
openodd |
The OpenODD files, relative to the binding file |
name, text |
The ODD's name and description |
probes |
Concept → probe (built-in name or package.module:function) or measure (a scenario measure's key, see Scenario measures), unit (built-in probes and measures know theirs), buckets (values, buckets, or range + every), text, and the coverage criteria target, cover_by, min_stay |
situations |
Module → its criteria (target, cover_by, min_stay, all optional): the modules to cover as situations (see Situations above) |
situations:
three_lane_cruise: {cover_by: meters, target: 200, min_stay: 3}
urban_junction_rain: {} # one stay of any length
Situations live in the binding file, not in the OpenODD documents: OpenODD has
no notion of coverage, and a construct of its own in an OpenODD document would
not be compliant. A module named under situations and not under the ODD
section is not a root candidate.
When no buckets are given:
- a categorical gets one bucket per literal, and a boolean gets two;
- a number gets buckets at the thresholds the modules test it against.
"<= 60 km/h"makes[-inf, 60)and[60, inf], so every boundary the ODD draws is tested from both sides. Each bucket holds its lower edge, so[60, inf]holds 60, which<= 60lets in. That bucket is partly inside, so it stays a coverage target. With"< 60 km/h", it would be outside.
A categorical defined by expressions is measured when what its expressions read is. A concept with no probe is always missing, and is reported as monitored but not covered.
Picking the ODD of a run¶
The odd config key, or ScenarioQueue(odd=...) /
ScenarioRunner(odd=...), takes one of:
| Value | ODD |
|---|---|
default |
The built-in one (Coverage): every attribute, no modules. It is openodd's bundled default ODD (its files), measured by this framework's probes, so it has a revision like any ODD read from files |
| a name | One registered with register_odd(name, builder) in code, or by a package through the autoware_carla_scenario.odds entry point group (or openodd's openodd.odds) |
path/to/urban.yaml |
A binding file (or an OpenODD file on its own, measuring nothing) |
package.module:function |
A builder to import and call |
uv run scenario scenario=intersection_passing/left_turn odd=path/to/urban.yaml
uv run scenario scenario=intersection_passing/left_turn odd=my_package.odds:urban_odd
A package that ships its ODD registers it through the
autoware_carla_scenario.odds entry point group, or openodd's openodd.odds
when the ODD is not CARLA's alone (both are read; openodd.resolve_odd()
resolves names as resolve_odd() does, with files measured by its backend). The entry point names a
function that takes nothing and calls register_odd, the convention the
scenario and traffic-backend groups use. A private taxonomy can then plug in
without the framework knowing it:
Sampling scenarios from the ODD¶
Coverage says which situations of the ODD the runs so far have driven. The
sampler draws the settings of the next runs from the ODD: only combinations
the ODD admits, and, if asked, aimed at what is still uncovered. It is a
sweep, so it is run by scenario-expand and by the lanelet sweeper, and with
autoware.launcher each case gets a fresh Autoware (Autoware as the
Ego):
uv run scenario --multirun hydra/sweeper=lanelet_constraint \
scenario=intersection_passing/left_turn map=town10hd_opt \
+sweep.odd_sample.count=20 +sweep.odd_sample.strategy=coverage \
'+sweep.odd_sample.coverage_from=[outputs]'
A glob batch with --multirun draws each matched scenario's cases the same
way and runs them all in one queue (Batch Execution).
Scenarios that draw from one ODD go on from each other's draws, in the order
the batch lists them: with strategy: coverage a later scenario counts the
earlier ones' cases as covered and aims at the buckets and situations they
left (see How a case is drawn); with
strategy: uniform it draws from a random stream of its own rather than
repeat their values. Here cut_in/right's 3 cases aim at what
cut_in/left's 3 left uncovered:
uv run scenario --multirun 'scenario=cut_in/*' map=town10hd_opt \
+sweep.odd_sample.count=3 +sweep.odd_sample.strategy=coverage
scenario-expand on the same glob lists those same cases (without the
arguments the batch adds), each scenario going on from the earlier ones'
draws, as in the batch.
The same seed, ODD, knobs, coverage and scenarios, in the same order, give
the same cases, on any machine (openodd's sampler draws from ChaCha8 seeded
with the seed's SHA-256). Two ODDs are one when they are built alike (their
structure), however the command line names them (a registered name, a YAML
path); a scenario on another ODD starts afresh.
or in the scenario's YAML:
sweep:
odd_sample:
count: 20 # cases, a whole number >= 1; one per lanelet case when left out
seed: null # a whole number in 0..2147483647 (2**31 - 1); the run's (`seed`) when null, `DEFAULT_SEED` (0) when the run has none either; same seed, ODD, knobs and coverage: same cases
strategy: coverage # uniform | coverage | weighted (weighted when `weights` is given)
coverage_from: [outputs] # earlier runs' *_coverage.json (files or directories)
weights: null # module -> share of the cases, or a YAML file of them; see "Sampling by shares of the ODD"
odd: null # the ODD to draw from; the run's (`odd`) by default
knobs: {} # by ODD attribute, replacing the scenario's controls; see below
flaky: null # look for flaky cases; see "Finding flaky cases" below
tire_friction: true # draw each case's grip in the range its weather allows; see "Tire friction follows the weather"
visibility: true # draw how far each case's drivers see in the range its weather allows; see "Visibility follows the weather"
count and seed are whole numbers (20 or 20.0, not 2.7 or a
boolean), count at least 1 and seed from 0 to 2147483647 (2**31 - 1); any other value is
refused, naming the key, before the map is read or anything starts. So are
an odd or strategy that is not a name (false, 0, "") and knobs
that are not a mapping (false, a list): a field left out or null takes its
default, any other value is refused rather than read as none.
Each case is a list of Hydra overrides. A sweep that also has constraints
(or a route search) gives each drawn case one of its lanelet cases -- in turn
unless the map decides some of the ODD's attributes, as it does the default
ODD's junction, lane count and speed limit: then each drawn case goes on a
lanelet case in the buckets it drew, so a lanelet case may run more than
once or not at all (The road a case is on). One
without draws from the ODD alone, and needs no map to expand. The last
override, ++odd_case=..., is the case itself, encoded: the run records it,
with its outcome, in its coverage file (see Finding flaky cases).
A drawn case's settings are the sweep's to choose, so the run's own
overrides may not set them: environment.time_of_day=night on the command
line of a sweep that draws the time of day is refused, naming the key and
the ODD attribute that draws it, before any case runs -- by scenario-expand,
a --multirun batch (local or submitted to SLURM), the lanelet sweeper and
the UI's Run panel alike. Left to Hydra, the override would win at run time
while the case recorded the label drawn, its sun and other weather effects
followed that label, coverage counted its bucket and the flaky search learnt
from a value the run never had. To hold an attribute, give it a knob in the
sweep that draws only what you want -- knobs: {<attribute>: {key:
environment.time_of_day, values: {night: night}}} for one label, a narrower
range for a number -- or narrow the ODD; then the case, its weather and
its coverage follow it. A CARLA weather field the conditions lead to (the
sun's altitude, the precipitation) and a scenario control's node are refused
the same way: change the rules they follow instead. Overrides of keys no
case sets (map=, ego.entity=) are not concerned, and one that names the
grip or the visibility, or where in its range it is, turns that draw off,
as a case that names one keeps it (Tire friction follows the
weather, Visibility follows the
conditions).
Who knows what¶
Three things meet in a sampled case, and none of them knows the others:
| Says | Knows | |
|---|---|---|
| The ODD | which conditions the system is for, and which scenario measure each of its attributes is (its mapping) | its taxonomy and the measure keys; no scenario |
| A scenario | its measures (BaseScenario's, or its own replacements) and its controls: which of its parameters sets which measure, and what it can stage |
the measure keys; not the ODD's taxonomy or bounds |
| The framework | the measure keys, and the weather and sun every scenario can be given | neither |
An ODD tightened, loosened or written in another taxonomy needs no scenario
changed; a scenario added needs no ODD changed. What a scenario sets up is
not a condition of the domain but an effect: the nodes it adds to the rules
(A scenario's nodes), drawn with every case whatever
the ODD names. A control without a quantity is still joined to the ODD on
the measure key: an attribute the ODD maps onto measure k is drawn with the
running scenario's control of k, and controls of measures the ODD does not
map are unused. A control with a quantity is never an attribute's knob,
in a Python ODD or a binding file alike (CarlaBackend.knob): an ODD that
maps an attribute onto its measure (headway: {measure: vehicle_ahead_gap_m,
buckets: [0, 10, 20, 40]} beside cut_in/left) draws the control as the
scenario's effect, and the attribute measures what it drew; a case's
settings file the control's key under the control, not the attribute
(#2072). A sweep's knob for that attribute (sweep.odd_sample.knobs)
replaces the control, as an explicit knob replaces any control: the
attribute is drawn with the knob, and the control is not drawn as an
effect. Attributes nothing sets are left open: a combination is
admitted when the ODD can hold for some value of them.
A scenario's nodes¶
A scenario adds nodes of its own to the shared rules (the
weather rules): what it sets up -- the gap to a vehicle
about to cut in, how fast a pedestrian crosses -- as a rule document under
scenario_rules in its config, version 3, joined to the shared network
(openodd's Backend.add_rules, its docs/rules.md "A scenario's nodes"). A
control that names a node by quantity writes it; a sweep draws every node
with every case, at a quantile of its own the flaky search can move, and the
case records the value it drew (case.quantities), which openodd's case plot
places the runs by:
# scenario/cut_in/left.yaml
scenario_rules:
version: 3
taxonomy: {bundled: default}
basis: {kind: expert, by: autoware_carla_scenario}
quantities:
vehicle_ahead_gap:
parts:
staged:
uniform: {low: 2.0, high: 30.0} # what the scenario can stage
vehicle_ahead_relative_speed:
parts:
staged:
uniform: {low: -15.0, high: 15.0}
controls:
vehicle_ahead_gap_m:
key: scenario.npc_gap_m
target: measure # the gap the measure reads at the start
quantity: vehicle_ahead_gap
vehicle_ahead_relative_speed_kph:
key: scenario.npc_initial_speed_kmh
quantity: vehicle_ahead_relative_speed
offset: ${ego.initial_speed_kmh} # its speed less the ego's
limits: [0.0, null] # a speed: never below zero
A node without at is a root: nothing in the domain conditions it yet. One
may read the shared nodes -- a gap that lengthens on a wet road reads the
grip -- and may not redefine one: the join refuses a node both define, so a
scenario can never change what another's cases follow. Each scenario's nodes
are its own: the cases of one never draw another's. Nor may a scenario name a node
tire_friction_quantile or visibility_quantile: a sweep adds those to its
nodes to draw where each case's grip and visibility are (see Tire friction
follows the weather).
What the run staged. A control is keyed by the measure it sets, so that
measure reads, in the run, the node the case drew: a run of a drawn case
measures each node its case drew by its control's measure -- the
vehicle_ahead_gap by vehicle_ahead_gap_m, the crossing_pedestrian_speed
by crossing_pedestrian_speed_ms -- on every tick of its first 10 s, beside
the ODD and counting toward no grade (openodd's CoverageCollector(effects=...),
written as effects.samples in the coverage file). openodd's drift view then
sets what the case drew against what the run staged, run by run: a gap placed
2 m off, or a relative speed the ego's first tick of acceleration moved, shows
as drift, and its correction is the control's scale and offset (openodd's
docs/drift.md, "Effects: what a scenario sets up"). A run no sweep drew
measures none.
Controls¶
A control (an OddKnob) ties a measure to the config key that sets it.
A scenario declares its own in its config, next to the parameters they set,
keyed by measure (A scenario's nodes has the
example scenarios' own).
| Field | Meaning |
|---|---|
key |
The config key the value is written to |
quantity |
A node of the rules (the scenario's scenario_rules, or the shared network's) whose value is written, drawn with every case: the control is then the scenario's effect, not an ODD attribute's knob, and range and values are not used |
target |
setting (the default, as is null): the key is a raw setting, written through scale and offset. measure: the key takes the measure's value at the start itself, written unchanged -- no scale, offset, integer or values (see below) |
range |
Numeric: [low, high] in the measure's unit, the values the scenario can bring about. A bucket is drawn only where it meets the range (by more than an edge), and only that part of it |
values |
Per bucket label. Categorical: the value to write, or [low, high) to draw it from ([x, x] is x); a bucket without one is not drawn. Numeric: a range to draw from instead of the bucket's interval -- which an unbounded bucket (-inf/inf edges) needs to be drawn at all |
scale, offset |
A numeric value drawn in the unit is written as value * scale + offset (scale not zero) |
integer |
Write an integer. A rounded numeric value that falls in another bucket is drawn again |
limits |
Numeric: [low, high] the written value must lie in, null for an open end -- what the key takes, whatever offset comes to. A bucket only values outside it would reach is not drawn |
Checking controls¶
scenario-check validates controls: of every config it checks, and
scenario-expand, a sweep and a --multirun batch refuse invalid ones -- a
concrete scenario's, without a sweep:, as well as a logical one's -- each
problem as one line with its config path, not a traceback:
cut_in/left: controls.vehicle_ahead_gap_m.range: expected [low, high] with low <= high, got [30, 2]
cut_in/left: controls.vehicle_ahead_gap_m.rnge: unknown field (did you mean 'range'?)
cut_in/left: controls.vehicle_ahead_gap_m: expected a key or a mapping, got 5
cut_in/left: controls.vehicle_ahead_gap_m.key: 'scenario.npc_gapm' is not a key of the config (did you mean 'scenario.npc_gap_m'?)
Errors (exit status 1; the config is not compiled): a control that is not a key
or a mapping, an unknown field, a missing or unknown key (it must be a key
of the composed config), a target that is neither setting nor measure,
scale/offset/integer/values under target: measure, a range or
limits that is not [low, high] with low <= high, a quantity that is
not a name, an integer that is not true or false ("false" and 0
are refused, not read as true and false), values that are not a mapping,
a scale or offset that is not a number. A field given null is one not
given: its default.
The knobs of sweep.odd_sample.knobs are checked the same way, before the
ODD is resolved, each problem naming its full path
(sweep.odd_sample.knobs.<attribute>.<field>: ...) in a ValueError; so
are the shares of an inline weights (#2226).
A measure name that is not a built-in one is only a warning
(controls.vehicle_ahead_gap: not a built-in measure (did you mean
'vehicle_ahead_gap_m'?)): a scenario may register measures of its own at run
time (register_measure), which a check of the config cannot see, so it
cannot tell a typo from such a measure.
{measure: key} is short for {measure: {key: key}}. The ODD's attribute
and the measure share a unit: the binding takes the measure's.
Raw settings and measure-targeted controls¶
A raw setting (target: setting) is a scenario parameter the drawn value is
mapped onto by scale and offset, and the mapping is the author's claim
that the parameter brings the measure about. It holds where the two are the
same quantity by definition. It is a guess where they are not:
npc_ahead_m is the NPC's s along its lanelet less the ego's along its
own, while vehicle_ahead_gap_m is bumper to bumper along the ego's lane.
Neighbouring lanelets do not run alike on a curve -- the outer one is longer,
and their s do not line up -- and the bumpers depend on the vehicles, so no
constant offset is right wherever the scenario is placed (on CARLA 0.10
Town04_Opt one guessed +11 m was measured 2.2 m off, and the right offset
there is right nowhere else).
A measure that the initial placement alone determines can be the
parameter instead. The scenario takes "the measure should read v at the
start" and solves for the placement that makes it so, evaluating the
measure's own definition on the poses it will spawn at; the control says
target: measure, and the value drawn is written unchanged:
from autoware_carla_scenario import place_at_vehicle_ahead_gap, vehicle_bumpers
# The NPC's s on its lanelet at which vehicle_ahead_gap_m reads cfg.npc_gap_m,
# seen from where the ego spawns -- whatever the two lanelets' shapes.
ego_at = to_carla_world(self._spawn_pose).to_carla_transform()
npc_pose = place_at_vehicle_ahead_gap(
ego_at,
cfg.npc_lanelet_id,
cfg.npc_gap_m,
ego_bumpers=vehicle_bumpers(world, self.ego_config.vehicle_type),
npc_bumpers=vehicle_bumpers(world, cfg.npc_vehicle_type),
)
place_at_vehicle_ahead_gap reads the gap with the measure's own code
(measures.vehicle_ahead_gap_between, which the measure reads every vehicle
with) for the NPC at each s of its lanelet, as a Lanelet2Pose spawns
there -- on the centreline, heading along it -- and narrows the first
crossing of v down by bisection (solve_along, a 1-D root find that takes
any s -> measure function, for a measure of a scenario's own). It solves
on the NPC's own lanelet and projects the NPC onto the ego's lane, so it does
not matter where the lanelets beside each other start or how long they are.
A gap the lanelet cannot reach -- too short, turning away, or out of the
lanes the measure counts -- is refused with the gaps it does reach, so a
drawn case that cannot be staged fails at setup rather than running another
gap.
The gap is bumper to bumper, so the placement needs both vehicles' sizes
before either is spawned. CARLA gives a blueprint no size: vehicle_bumpers
reads the bounding box of a vehicle of the blueprint already in the world
when there is one, and otherwise spawns one high above the map (higher again
if that spawn is refused), reads its bounding box and destroys it before the
world ticks again -- once per blueprint per process. The vehicle-ahead
measures do not count a vehicle that far above the ego. The NPC is then spawned once, where the gap reads right, rather
than spawned, measured and moved -- a move that would race its initial
speed and the first tick.
Which to use:
| The measure is | Control |
|---|---|
decided by where and how road users start, and the parameter is that measure by definition (npc_initial_speed_kmh less the ego's is the relative speed) |
a raw setting, with the offset that makes it the measure |
| decided by where road users start, but the parameter is another quantity (an arc length for a gap) | target: measure, with the scenario solving its placement |
| decided by what the run then drives (a pedestrian's gap when it sets off, anything a controller or traffic changes) | a raw setting; how far the measure drifts from what was drawn is read off the runs, in openodd's case plot (its drift/calibration view), not solved at setup |
The relative speed of cut_in is the first kind: both vehicles start at
their speeds along their lanes, and the measure is the NPC's velocity along
the ego's lane at the NPC less the ego's along it at the ego, so its
offset: ${ego.initial_speed_kmh} writes it exactly: on a bend the NPC
heads along its lane, which runs alongside the ego's lane where it is. (A
lane beside a bend is shorter or longer than the ego's, so an NPC on it
covers the ego's lane a few per cent slower or faster -- 3.5 m off a 100 m
radius, 3.5 % -- which the measure does not count: it reads speeds, not how
fast s changes.)
The weather and the sun are set for every scenario, through the
environment config, which sets them before the run starts
(EnvironmentAction). The default ODD names the conditions -- the
rainfall rate, the visibility through the fog, the time of day, the wind
speed and the cloud cover, the taxonomy's -- and the built-in probes
reading them have controls of their own:
| Probe | Key | Range |
|---|---|---|
rainfall_rate_mm_h |
environment.rainfall_rate_mm_h |
0-30 mm/h, within the bucket drawn (the rain intensity's: light 0.1-2.5, moderate 2.5-7.6, heavy above) |
fog_visibility_range_m |
environment.fog_visibility_range_m |
15-5000 m, within the bucket drawn (the fog level's: thick under 50, heavy 50-450, medium 450-800, mist 800-1600) |
time_of_day |
environment.time_of_day |
the label drawn: day, dusk, night, dawn |
wind_speed_kmh |
environment.wind_speed_kmh |
0-100 km/h, within the bucket drawn (the wind intensity's: light 5-20, moderate 20-50, strong above) |
cloud_cover |
environment.cloud_cover |
the label drawn: clear, partly_cloudy, overcast |
What CARLA cannot bring about -- snow and the temperature, which the
default ODD's taxonomy declares -- the CARLA backend names
(CarlaBackend.unsupported, odd.backend.UNSUPPORTED_CONCEPTS): an ODD that
measures one, or whose modules restrict one or a concept defined by one (the
snow intensity), is refused as it is read, before anything runs
(openodd.UnsupportedConceptError), rather than covered by runs that never
rendered it. The default ODD declares them and measures neither.
CARLA does not take these: it renders them by its own weather fields, and
which fields render a condition is what the weather rules
say -- a Bayesian network from the conditions to CARLA's precipitation,
cloudiness, precipitation_deposits, wetness, fog_density,
sun_altitude_angle and sun_azimuth_angle, and on to the grip and the
visibility. They are worked out once, before the run, and the run applies
numbers: a sweep.odd_sample draws each field that follows from a condition
the ODD names with the case, at a quantile of its own (an effect,
environment.precipitation=... among the case's overrides, its value in the
case's quantities, which the flaky search moves); a config that names a
condition gets the rest at its median
(weather_rules.resolve_environment). A CARLA field the config sets itself
is kept, and what follows from it in the rules reads it. The probes read the
conditions back: the rainfall rate, the fog visibility, the wind speed and
the cloud cover as the run was set to them, the time of day off the sun.
The cloud cover is the sky's: rain clouds CARLA's sky past it (the rules'
carla_cloudiness is the cloudier of the sky and the rain's clouds), so a
case drawn clear in moderate rain renders a cloudy sky and is measured
clear, the cell it was drawn for. A run that names no cloud cover is
measured off CARLA's cloudiness.
A config that sets CARLA's fields alone names no condition: the ODD's weather is not measured, and drivers see as they always did (a warning says so). CARLA's own fields have controls too, for an ODD that names them:
| Probe | Key | Ranges |
|---|---|---|
rain |
environment.precipitation |
none 0, light 1-30, moderate 30-70, heavy 70-100 |
fog |
environment.fog_density |
as rain |
illumination |
environment.sun_altitude_angle |
day 15-90, low_sun 0-15, twilight -6-0, night -90 to -6 |
visibility |
environment.visibility_m |
15-400 m, whatever the weather |
road_wetness |
environment.wetness |
as rain |
tire_friction |
environment.tire_friction |
0.3-1.1 of the dry road's grip, whatever the weather |
wind |
environment.wind_intensity |
as rain |
cloudiness |
environment.cloudiness |
as rain |
The precipitation brings, by the rules, the cloudiness, standing water and
wetness of CARLA's rain presets: precipitation alone draws only the falling
drops, and a camera sees a dry road under a clear sky. road_wetness reads
the weather's wetness alone, which its knob sets, so a drawn value reads
back as its bucket; the grip follows the wetter of it and the standing
water. tire_friction reads the grip the run's weather left the vehicles'
wheels (a share of the dry road's, 1.0) -- None when nothing set it -- and
its knob names it outright, as visibility's names the visibility. wind
and cloudiness read CARLA's 0-100 intensities as levels, as rain and
fog do.
An ODD that names a condition and the CARLA field that follows from it
-- the taxonomy's rainfall_rate beside rain or road_wetness,
the cloud cover (cloud_cover) beside cloudiness, wind_speed beside
wind, fog_visibility_range beside fog -- has the field drawn by the
field's knob, not by the rules: the knob wins, and no effect writes that
field or any field that follows from it in the rules (the precipitation's
cloudiness, standing water and wetness), so every bucket a case records is
the one the run reads back (#2072). The runner works those fields out from
the value the knob wrote, at their median
(weather_rules.resolve_environment, and EnvironmentAction's road water,
in which a field set wins), as it does for a config that sets the field
itself; the case's quantities and the flaky search hold only the effects
left. illumination beside time_of_day is refused when the cases are
drawn: its knob sets the sun's altitude, which the time of day is read back
from, so no run could keep the buckets drawn for both (ValueError: ODD
...: environment.illumination sets environment.sun_altitude_angle with its
knob, which ...time_of_day is read back from ...); an ODD names one of them.
Every environment.* value must be a finite number -- an integer or a
float, not a boolean or a string (true and "15" are refused, not read as
1 and 15) -- but time_of_day, one of day, dusk, night, dawn, and
cloud_cover, one of clear, partly_cloudy, overcast: any
other value, NaN or infinity is refused before the run starts, naming the key
(environment.rainfall_rate_mm_h must be finite, got nan). cloudiness,
precipitation, precipitation_deposits, wetness, wind_intensity and
fog_density must be within 0-100, tire_friction, visibility_m and
fog_visibility_range_m positive, fog_distance, rainfall_rate_mm_h and
wind_speed_kmh not negative, sun_altitude_angle within -90 to 90 degrees
and sun_azimuth_angle within 0 to 360 (CARLA's ranges; a value outside is
refused, not wrapped or clamped), and the two quantiles within 0-1. EnvironmentAction refuses the same values with a
ValueError naming its argument.
Fog is drawn as fog_density alone. How thick that looks is the server's: a
CARLA 0.10 server on the UE 5.8 engine (the ue58-dev tree) multiplies it by carla.Weather.FogDensityScale,
whose default 0.001 leaves even fog_density=100 a faint haze on the horizon.
Start it with -ExecCmds="carla.Weather.FogDensityScale 0.01" for fog a
camera sees, or set server.fog_density_scale=0.01
(LiDAR).
The precipitation and the fog density set the LiDAR noise as well: each
driver.lidars sweep carries the world's precipitation and fog density
(LiDAR).
A scenario that sets the weather itself has the last word over the config:
its own EnvironmentAction, with numbers -- worked out from conditions with
EnvironmentAction(**resolve_environment({"rainfall_rate_mm_h": 5.0})).
Tire friction follows the weather¶
CARLA draws rain but does not drive on it: a vehicle grips a soaked road as
it grips a dry one. So an EnvironmentAction that sets the rain or the
road's water -- wetness or precipitation_deposits, which rain brings --
also scales every
vehicle's wheel friction (friction_force_multiplier) to a share of the dry
road's grip, within the range the resulting water leaves
(tire_friction_range). The default weather rules give it
by the wetter of wetness and precipitation_deposits:
| Road water | Grip, share of the dry road's | The rainfall that leaves it |
|---|---|---|
| 0 (dry) | 0.90-1.10 | none |
| 50 (wet) | 0.65-0.85 | moderate, about 4 mm/h (standing water about 50) |
| 100 (flooded) | 0.45-0.65 | heavy, 25 mm/h and above |
Between the rows the bounds are interpolated (the rules write the distribution at these points). Measured tire-road friction is about 0.8-1.0 on dry asphalt, 0.5-0.7 on wet and 0.3-0.5 under a water film; each range spans what tires, their wear and the asphalt make of the same water, and a dry road's is centred on CARLA's own grip, so a dry run drives as it did before.
Rain brings the water with it however it is set. An EnvironmentAction
that sets precipitation and not wetness or precipitation_deposits --
a scenario's own, or the editor's "Set Weather" card -- sets those to what
the rules give that rain, at their median (weather_rules.road_water), as
the config's environment does before the run: precipitation=100 leaves a
wetness of 97.5, standing water of 90 and a grip of 0.56 either way. Rules
whose water reads a condition read the run's: the rainfall rate, fog
visibility, wind speed or cloud cover the action names, else the one the run
was set to (the config's environment, or an earlier action), and the time
of day the sun renders once the action has set it. A wetness or
precipitation_deposits the action gives wins.
So stopping the rain dries the road at once: under the default rules,
EnvironmentAction(precipitation=0) sets wetness and
precipitation_deposits to 0 and the grip back to the dry road's, as a
config's precipitation: 0 does. CARLA does not let a road dry over time;
a scenario whose road should stay wet after the rain gives the water
itself, EnvironmentAction(precipitation=0, wetness=60,
precipitation_deposits=30).
Besides the water the rain brings (above), the grip is the one quantity the
action reads off the rules during the run:
it may differ by vehicle (Rules by vehicle), and a vehicle
spawned later gets its own. It reads every CARLA field of the weather the
action leaves -- the water the run's conditions gave before it started --
and the conditions themselves, as the ODD's probes read them: the rainfall
rate, the visibility through the fog and the wind speed the run was set to,
the cloud cover it was set to (else the one CARLA's clouds render), the time
of day CARLA's sun renders, and how far drivers see. So a tire_friction
part may read a cause or any other quantity; a rainfall rate, fog
visibility or wind speed the run does not name reads its default, as it
does before the run, while the time of day and the cloudiness are always
read off CARLA's sun and clouds (the cloud cover the run names wins).
Under rules other than the default ones -- a rule file of the run's
(environment.rules) or rules by vehicle -- the grip may differ from
CARLA's on any road, so every EnvironmentAction sets it, whatever else it
sets, and the config's environment registers one even when it names only
rules. One that names no grip, quantile or water (a scenario's own that
sets the sun, say) reads the grip again at the quantile the run's grip is
at, and keeps a grip named outright (tire_friction). A grip named
outright also ends the quantile: water set after it reads the grip at the
median. Under the default rules, a run that sets no road water, no quantile
and no grip keeps CARLA's grip (1.0, whatever water CARLA renders).
Where in the range a run's grip is, is environment.tire_friction_quantile:
the quantile of the grip's distribution (0 its low end, 1 its high end; the
median when not set). A later action in the run that names none -- rain
starting mid-run -- reads the grip at the same quantile. A
sweep.odd_sample draws it for every case (environment.tire_friction_quantile=0.412
among the case's overrides, recorded in its coverage file) as a parameter of
the case, quantile(tire_friction_quantile) among its values -- a root node
of the rules, uniform on [0, 1], drawn as the weather effects' quantiles are
-- so the same sweep draws the same grip, and the flaky search sees and
moves it like any other parameter. The weather comes
from the ODD and the grip from the range that weather allows: a heavy-rain
case never runs on a dry road's grip. tire_friction: false in the
odd_sample leaves every case at the median -- of the rules' grip where the
case sets it, as above; the dry road's, CARLA's own, where it does not --
(true or false, or one of
their spellings, as the run's switches take: "off",
"no" and 0 are false; a missing or null one is true; anything else, such
as ture or 2, is refused, naming sweep.odd_sample.tire_friction, before
any case is drawn), and
environment.tire_friction fixes the grip whatever the weather (it wins over
the quantile, and a case that names one draws no quantile; the ODD's
tire_friction probe draws it through that knob). A run whose own overrides
name the grip or its quantile draws none either.
The vehicles in the world when the action fires are scaled then; a scenario
with such an action -- the config's, or its own -- also runs
follow_tire_friction after every tick, which gives the vehicles spawned
later (background traffic) the same grip. A scenario that sets the weather on
the tick loop sets the grip with it: a factor applied again replaces the last
one rather than compounding it.
The grip is CARLA's vehicle physics alone: no traffic backend reads it.
The example scenarios declare these, each a node of their scenario_rules:
| Scenario | Measure | Node | Parameter |
|---|---|---|---|
cut_in/* |
vehicle_ahead_gap_m |
vehicle_ahead_gap, 2-30 m |
npc_gap_m (target: measure) |
cut_in/* |
vehicle_ahead_relative_speed_kph |
vehicle_ahead_relative_speed, -15 to 15 km/h |
npc_initial_speed_kmh, offset by the ego's |
pedestrian_dart_out |
crossing_pedestrian_gap_m |
crossing_pedestrian_gap, 8-30 m |
trigger_distance_m |
pedestrian_dart_out |
crossing_pedestrian_speed_ms |
crossing_pedestrian_speed, 0.8-3 m/s |
walk_speed_ms |
traffic_light_compliance declares none, but has what a control needs:
stop_line_distance_m (target: measure: the ego is moved along the lane its
spawn pose is on until stop_line_distance_m reads it,
place_at_stop_line_distance), and its own measure red_light_duration_s,
the seconds every light is held red (light_switch_delay_seconds, a raw
setting). With the delay drawn, timeout_after_green_seconds counts the
fail-safe timeout from the green light instead of from the start. A drawn
distance does not decide the verdict by itself: the run is judged by where
the ego stops against that same line -- not past it on red, standing in the
stop zone (stop_zone_min_m-stop_zone_max_m before it) by the green -- so
an ego started far back may roll up to the line on red.
sweep.odd_sample.knobs takes the same fields keyed by the ODD's attributes,
and replaces anything above, attribute by attribute, for that sweep. It is for
the command line -- one sweep against one ODD -- not for a scenario's config,
which would then name the ODD's taxonomy: a scenario says controls. Controls set where a run starts;
the measures read what it then drives, which is what coverage counts.
Visibility follows the conditions¶
CARLA draws the weather but its traffic never looks through it: the
TrafficManager's vehicles know where every actor is, fog or not. A driver
does not, and drives so as to stop within the road they see. So how far
drivers see is a quantity of the run, in metres, which the conditions bound
(the weather rules' visibility). Each of the fog, the
rain and the light gives a range, and the shortest holds:
| Condition | Visibility, m |
|---|---|
| clear day | 250-400 |
| fog visibility 15 / 50 / 150 / 400 m and over | 12-18 / 40-60 / 120-180 / 250-400 |
| rainfall rate 2.5 / 7.6 / 25 mm/h | 180-320 / 100-220 / 60-150 |
| dusk, dawn | 150-275 |
| night | 50-150 (a dipped beam to a main beam or a lit street) |
Between the rows the bounds are interpolated. It is worked out before the
run, from the conditions the run names, at environment.visibility_quantile
of its distribution (0 its low end, 1 its high end; the median when not
set), and the run records it as environment.visibility_m, which also fixes
it whatever the conditions. A sweep.odd_sample draws the quantile for every
case, as it draws the grip -- a parameter of the case,
quantile(visibility_quantile), which the flaky search moves -- so the ODD's rainfall rate, fog visibility and
time of day are what shorten it: a night case's drivers see 50-150 m, a
thick-fog case's 12-18 m. visibility: false in the odd_sample leaves every
case at the median (read as tire_friction is,
strictly). An ODD that names the visibility itself
(the visibility probe) sets it through its own knob,
environment.visibility_m, and a case with one keeps it, as does a run
whose own overrides name the visibility or its quantile. A run that names no
condition records none, and its traffic sees as it always did; each run
starts with none, whatever the last left.
What follows it is the traffic backend's (traffic.vehicles.follow_visibility):
the sumo backend holds its own
traffic to the speed its drivers keep in it -- the weather rules'
sight_speed, which reads the visibility -- by default
(Visibility); the traffic_manager
backend holds every vehicle it drives to it with follow_visibility: true,
under its road's speed limit
(Weather and the TrafficManager),
and otherwise only logs the speed it would hold traffic to.
Weather rules¶
What the conditions do is a Bayesian network of conditional distributions,
not code: weather_rules.yaml in the package
(autoware_carla_scenario.weather_rules), the CARLA backend's own rules
(CarlaBackend.rules), which openodd reads and evaluates in its Rust core
(its rules docs
have the file format in full). It is a version 3 file:
- its causes are its inputs, concepts of the taxonomy it names
(
taxonomy: {bundled: default}, openodd's bundled subset of tier4/taxonomy):rainfall_rate(mm/h),fog_visibility_range(m),time_of_day(a label),wind_speed(km/h) andcloudiness(a label) --weather_rules.WEATHER_INPUTS, which theenvironmentkeysrainfall_rate_mm_h,fog_visibility_range_m,time_of_day,wind_speed_kmhandcloud_covergive (weather_rules.CAUSE_SETTINGS); - its effects are its quantities, in no taxonomy: CARLA's weather fields
(
carla_precipitation,carla_cloudiness,carla_precipitation_deposits,carla_wetness,carla_wind_intensity,carla_fog_density,carla_sun_altitude,carla_sun_azimuth:weather_rules.CARLA_QUANTITIES), the grip (tire_friction), the visibility and the speed a driver keeps (sight_speed); - every part says what it rests on (
basis): the default rules are an expert's, set from CARLA's presets and published ranges, not fitted.
A quantity is one or more parts; each reads some causes or other quantities and gives the quantity's distribution at points of them:
quantities:
carla_precipitation:
parts:
rain:
at: {rainfall_rate: [0, 0.1, 2.5, 7.6, 25]}
uniform:
low: [0, 1, 25, 60, 95]
high: [0, 10, 35, 75, 100]
tire_friction:
combine: min
parts:
wetness:
at: {carla_wetness: [0, 50, 100]} # dry, wet, flooded
uniform:
low: [0.90, 0.65, 0.45]
high: [1.10, 0.85, 0.65]
standing_water:
at: {carla_precipitation_deposits: [0, 50, 100]}
uniform: {low: [0.90, 0.65, 0.45], high: [1.10, 0.85, 0.65]}
- A part's distribution is
uniform(low,high),triangular(low,mode,high) ornormal(mean,sd, cut tolow..high). Each parameter is one number or one value per point; between number points it is interpolated linearly, beyond them it holds the outermost point's value. A categorical cause (the time of day) is written at its labels, with nothing between them. So the default rules give exactly the tables above. - A part may read several inputs, with a value at every combination of their points, as nested lists.
- A run's value is the one at its quantile: the value with that share of
the distribution below it (
weather_rules.weather_at). With more than one part, every part gives its value at the same quantile andcombine(min, the default, ormax) joins them: the visibility's fog, rain and light are three parts, and the shortest holds. The parts move together -- a case at quantile 0.1 is at the worse end of the fog's, the rain's and the light's alike -- so the quantity's range is the least of the parts' lows to the least of their highs. - A cause the run does not name reads its
default: no rain, no fog (10 km), the day.weather_rules.resolve_environmentworks out only the fields that follow from what the run names, so a run that names the time of day alone keeps the map's rain.
Chains. A part may read another quantity, so the rules chain: the
rainfall rate gives CARLA's precipitation, which gives its wetness and
standing water, which give the grip, and its cloudiness -- the cloudier of
the sky the run names and what the rain brings (combine: max); and sight_speed, the speed (m/s) a
driver keeps, reads the visibility. Its distribution is triangular at each
visibility: at most the speed SUMO's default car (braking at 4.5 m/s² a
second after it sees) stops from within it, v * tau + v² / (2 * decel) =
the visibility, and down to three quarters of that
(visibility.sight_speed).
| Visibility, m | 15 | 50 | 100 | 250 | 400 |
|---|---|---|---|---|---|
sight_speed, km/h (low / mode / high) |
22 / 27 / 29 | 46 / 59 / 62 | 70 / 88 / 93 | 117 / 148 / 155 | 150 / 190 / 200 |
visibility.sight_speed_at(visibility_m, quantile, decel, reaction_s) reads
it at a quantile for a car that brakes and reacts otherwise: the rules'
value scaled by that car's stopping speed over the default car's, and never
above that car's stopping speed. A traffic backend holds each of its
vehicles to it at a quantile of the vehicle's own
(visibility.driver_quantile, from its id: the same in every run, spread
over the traffic), so some drivers keep close to what they can stop from and
others well below it. weather_rules.default_rules().sample(conditions)
gives every quantity of the conditions at once, each at its median.
A rule file is checked when it is read: every field known, every input's
points increasing, one value per point, the distribution valid at every
point (low not above high, a mode between them, a positive sd), and
every input a part reads a concept of the taxonomy (openodd.RulesError
says where). A version 2 file still reads merged over the default rules (it
keeps their version and taxonomy), but one that reads an input standing for
no concept is refused.
A run's rules. environment.rules names a rule file -- a path, absolute
or relative to the directory the run starts in, or an https:// URL --
merged over the default rules: it names only what it changes (a part's
uniform without its at, say), and may extends other files, by path,
URL (pinned with {url, sha256}) or git source. A sweep sets it like any
other key (environment.rules=/data/grip_measured.yaml); every run sets the
rules it names, or the default ones, when it starts, so one run's do not
reach the next -- in a batch too, which builds every case before any runs:
each case runs, and records in its coverage file, its own rules.
Rules by vehicle. Tyres differ by vehicle; the weather does not:
environment:
rules:
default: weather/measured.yaml # every vehicle no entry names
vehicles: # the first entry a vehicle matches
ego: weather/ego_measured.yaml # the ego (role_name "Ego")
"vehicle.mini.*": weather/mini.yaml # a blueprint pattern
Each vehicle's file is read over the run's default rules. default and
every vehicles entry name one file each -- a path or an https:// URL, as
environment.rules itself does; a pinned {url, sha256} is for extends
alone, so a file to pin goes in one that extends it. Anything else (a
number, a mapping, a list, an unknown key) is refused before the batch runs
(openodd.RulesError, naming the key; weather_rules.check_weather_rules).
Every EnvironmentAction (the config's environment, even one naming only
rules, registers one) gives each vehicle the grip of its own rules, all at
the same tire_friction_quantile (a case draws one; the median when none is
set), and a vehicle spawned later its own too; the ODD's tire_friction
probe reads the ego's. tire_friction, the grip named outright, is every
vehicle's. Everything else -- CARLA's weather, the visibility -- is the
default rules'.
weather_rules.weather_rule_files() lists every rule file a run follows,
with the SHA-256 of each.
Traffic parameters from the ODD¶
How the traffic behaves -- how fast drivers go against the limit, how they react, how often they run a red light, how fast pedestrians walk -- is the run's traffic parameters (Traffic parameters). Each numeric one is a spread, a mean and a standard deviation, that every vehicle or pedestrian draws its own value from (a normal distribution cut to the parameter's bounds). An ODD bounds the spread: the mean and the std of each are built-in probes, with knobs, so an ODD that names them makes a sweep draw both per case, and every road user of the case then draws its own value from them.
| Probe | Knob (written) | Knob range |
|---|---|---|
vehicle_<field>_mean |
traffic.vehicles.<field>.mean |
the field's mean range below |
vehicle_<field>_std |
traffic.vehicles.<field>.std |
the field's std range below |
pedestrian_<field>_mean |
traffic.pedestrians.<field>.mean |
likewise |
pedestrian_<field>_std |
traffic.pedestrians.<field>.std |
likewise |
bicycle_<field>_mean |
traffic.bicycles.<field>.mean |
likewise (sumo only) |
bicycle_<field>_std |
traffic.bicycles.<field>.std |
likewise (sumo only) |
| Parameter | Unit | Mean range | Std range |
|---|---|---|---|
vehicles.speed_factor |
0.5-1.5 | 0-0.3 | |
vehicles.min_gap_m |
m | 0.5-10 | 0-3 |
vehicles.ignore_lights, ignore_signs, ignore_vehicles, ignore_walkers, random_lane_changes |
0-1 | 0-0.5 | |
vehicles.accel |
m/s^2 | 0.5-5 | 0-1.5 |
vehicles.decel |
m/s^2 | 1-9 | 0-2 |
vehicles.reaction_time_s |
s | 0.3-3 | 0-1 |
vehicles.imperfection |
0-1 | 0-0.5 | |
vehicles.max_lateral_speed_mps |
m/s | 0.2-2 | 0-0.5 |
vehicles.max_lateral_accel |
m/s^2 | 0.2-3 | 0-0.5 |
pedestrians.walking_speed_mps |
m/s | 0.5-2.5 | 0-0.5 |
pedestrians.min_gap_m |
m | 0.1-2 | 0-0.5 |
pedestrians.red_light_violation_s |
s | 0-60 | 0-30 |
bicycles.max_speed_mps |
m/s | 2-8 | 0-1.5 |
bicycles.accel |
m/s^2 | 0.5-2 | 0-0.5 |
bicycles.decel |
m/s^2 | 1-4 | 0-1 |
bicycles.min_gap_m |
m | 0.2-2 | 0-0.5 |
bicycles.reaction_time_s |
s | 0.3-2 | 0-0.5 |
bicycles.imperfection |
0-1 | 0-0.3 | |
bicycles.max_lateral_speed_mps |
m/s | 0.2-2 | 0-0.5 |
bicycles.red_light_violation_s |
s | 0-30 | 0-10 |
The bicycles' are traffic.bicycles (BicycleParameters,
Traffic parameters): only SUMO
simulates bicycles, so an ODD that draws them is an ODD for the sumo
backend.
A bucket of the ODD's attribute narrows the range, as for any knob. In a binding file:
# binding.yaml
behaviour.speed_factor: {probe: vehicle_speed_factor_mean}
behaviour.speed_spread: {probe: vehicle_speed_factor_std}
behaviour.reaction_time: {probe: vehicle_reaction_time_s_mean}
behaviour.walking_speed: {probe: pedestrian_walking_speed_mps_mean}
A case drawn with behaviour.speed_factor in [0.8, 0.9) and
behaviour.speed_spread in [0.05, 0.1) writes
traffic.vehicles.speed_factor.mean and .std, and each vehicle of its run
draws its own speed factor from that normal distribution. The probes read
what the run set, every tick, so the run's coverage records the mean and std
it drove with; a run that set none reads a missing value.
A parameter the run's backend does not support is still refused: an ODD that
draws a reaction time is an ODD for the sumo backend, and a case of it run
on the traffic_manager backend fails before it starts
(Traffic parameters).
The road a case is on¶
A sweep with cases of its own -- constraints, or a route search -- draws the
attributes the Lanelet2 map decides as well: whether the case is in a
junction, how many lanes, the speed limit, the location and road type
tags. Each is read off every case's lanelet (its first
*lanelet_id override: the constraints' pick, or the route's spawn) with the
probe's lanelet form, and gets a knob that reaches the buckets those lanelets
are in. The knobs write no setting a run reads (_odd_map.*, taken off
again); instead each drawn case is put on a case whose lanelet is in every
bucket drawn, the one run least so far among those that are. So a lanelet
case may run more than once, or not at all, even without a count: the
buckets decide, not the list of cases. Each run records its case
(++odd_case) with its own lanelet's buckets and values, not the ones
drawn.
Not every combination of buckets is on one lanelet -- no junction where the
limit is 30, say -- and each knob draws its attribute on its own, so 20
cases are drawn for each one kept and those no lanelet fits are passed over.
If too few fit, the rest go on the lanelet in most of their buckets, with a
warning. So strategy: coverage aims at the road as at everything else: the
buckets of junction, lane count or speed limit that earlier runs left uncovered
come first, as far as the sweep's cases reach them.
A sweep that gives an attribute a knob itself (odd_sample.knobs) keeps it.
The flaky search (odd_sample.flaky) draws its cases as it chooses them, none
passed over; each goes on the lanelet in most of its buckets.
How a case is drawn¶
- Each knob's attribute gets one of its buckets, leaving out the buckets
outside the ODD (the ones
scenario-odd showlists as out). - The combination is kept only if the ODD's verdict on those buckets is not outside -- so modules that rule out a combination (no heavy rain at night, say) are honoured, not just single buckets.
- A value is drawn inside each bucket (a numeric bucket's interval, or the
knob's range) and checked against the ODD again, as a value, with the
attributes an OpenODD ODD derives from others worked out: a condition finer
than a bucket (
between(0, 45)across a[30, 60)bucket) holds for what is drawn. What is written is the value exactly as drawn, and a numeric one is read back to check it is still in its bucket.
strategy: uniform draws every admitted bucket alike. strategy: coverage
reads coverage_from and draws each attribute from its least covered
buckets first, counting the cases already drawn in the batch -- by earlier
scenarios on the same ODD too, in a glob batch -- as covered, so a batch
works through the holes before it repeats one; it falls back to any
bucket (the less covered the likelier) only when the ODD admits no
combination of the least covered. Situations that are still holes (or
that no earlier run measured) are aimed at first, one case each; inactive
ones are not, nor are those an earlier case of the batch was aimed at. A
situation the knobs alone decide -- every attribute it tests has a knob, and it refers to no other module -- holds in its case. One that also tests what no knob sets -- a speed, a road -- gets a case
under which it can hold; whether it does is up to the drive. A situation
the ODD and the knobs leave no case for is given up on, with a warning.
coverage_from paths are relative to the directory the sweep is started
from. Coverage is read when the sweep is expanded: to resume a sweep
(--resume-from) into the same cases, read the same coverage -- point
coverage_from at the earlier runs, not at the sweep's own output.
From Python:
from pathlib import Path
from autoware_carla_scenario.coverage.report import load_and_merge
from autoware_carla_scenario.odd import OddKnob, OddSampler, resolve_odd
sampler = OddSampler(
resolve_odd("path/to/urban.yaml"),
controls={"vehicle_ahead_gap_m": OddKnob("scenario.npc_gap_m", range=(2.0, 30.0))},
seed=0,
strategy="coverage",
coverage=load_and_merge([Path("outputs")]),
)
for case in sampler.sample(20):
print(case.overrides, case.buckets, case.situation)
To go on from cases drawn before -- another scenario's, on the same ODD --
pass them as drawn; OddDraws keeps them per ODD, as a batch does:
from autoware_carla_scenario.odd import OddDraws
draws = OddDraws()
draws.add(sampler.odd, sampler.sample(20))
later = OddSampler(sampler.odd, seed=0, strategy="coverage", drawn=draws.of(sampler.odd))
Sampling by shares of the ODD¶
weights gives chosen parts of the ODD a share of the cases, and draws the
rest as strategy: uniform would (it implies strategy: weighted). The
parts are the ODD's modules, named in its OpenODD files, of one condition
or of several -- so the sweep names no attribute of the taxonomy, and a module
redefined in the ODD needs no change to the sweep:
# regions.yaml: parts of the ODD, in plain OpenODD
IMPORT: [urban.yaml]
MODULES:
heavy_rain:
INCLUDE_AND:
environment.rain: heavy
rainy_night_merge:
INCLUDE_AND:
environment.rain: heavy
environment.illumination: night
scenery.road_type: merge
# shares/rainy.yaml
heavy_rain: 0.3 # 30% of the cases in heavy rain
rainy_night_merge: 0.1 # 10% in heavy rain at night in a merge
uv run scenario --multirun 'scenario=cut_in/*' odd=path/to/urban.binding.yaml \
+sweep.odd_sample.count=50 +sweep.odd_sample.weights=shares/rainy.yaml
# or inline, for one module
'+sweep.odd_sample.weights={heavy_rain: 0.5}'
The binding file names the regions file with the ODD's
(openodd: [urban.yaml, regions.yaml]), and the ODD names its roots under
ODD:: a module that is a root -- a document with no ODD: section takes
every module that is not a situation as one -- would bound the ODD, and is
refused a share (every case is in it). A relative path to a file of shares
is resolved against the working directory the command is started from (a
--multirun, a batch, scenario-expand), not the directory of the scenario's
config, as coverage_from is: the path is given on the command line, or in a
config the command line composes. Give an absolute path for a file next to a
scenario package. A file that cannot be read or parsed, or that names a module
the ODD does not have, refuses the sweep before any case runs, with one line
Error: <scenario>: <why> and exit 1.
Each module holds its share of the cases, however the modules overlap: with
heavy_rain: 0.3 and night: 0.3, 30% of the cases are in heavy rain and
30% at night, those in both counting for both. A share is a fraction in
[0, 1], not a weight relative to the others. Inside and outside the modules
the cases are drawn as the uniform strategy draws them (of all the ways to
hold the shares, the one closest to it), rounded to whole cases counting the
cases drawn before -- by the earlier scenarios on the ODD in a glob batch too,
so ten scenarios drawing 1 case each with heavy_rain: 0.3 draw 3 in heavy
rain between them.
Refused: a name that is no module, a root, a share outside [0, 1],
weights with another strategy; a module whose truth the knobs do not
decide (it tests an attribute nothing draws, such as the ego's speed); shares
that cannot hold together inside the ODD (a: 0.7 and b: 0.6 for modules
that never overlap). With flaky, the part of each batch the search does not
explore holds the shares. A sweep with constraints whose map decides some
attributes keeps, of more cases drawn, those on a lanelet of the sweep, so
its cases hold the shares only roughly. The shares are the sampler's, not OpenODD's, which
has no construct for a distribution (openodd's
weighted sampling).
Finding flaky cases¶
Every run a sweep draws from the ODD records its case in its coverage file:
the values it ran with, why it was drawn, and whether it passed (openodd's
CoverageCollector.record_case and record_outcome; the runner calls them,
so a scenario needs nothing of its own). The merged report then lists the
runs, flakiest first, with their flakiness assessed over all the runs of
their scenario (scenario-coverage, section Flaky cases).
With flaky, the sweep also looks for them. openodd's FlakySearch learns
where the runs in coverage_from passed and failed (Gaussian-process
classification), and draws part of each batch where a small change would flip
the outcome -- after the sampler has covered every bucket, the rest of the
batch still sweeping the ODD (openodd's
docs/flaky.md):
sweep:
odd_sample:
count: 20
strategy: coverage
coverage_from: [outputs] # the earlier sweeps' runs: what the search learns from
flaky:
explore_fraction: 0.8 # of each batch, once every bucket is covered
scenario: CutInScenario # learn from this scenario's runs only (all, when left out)
flaky: true takes the defaults; false, null or leaving it out is no
flaky search. A switch is read as tire_friction's is, so "true", 1,
"false" and 0 mean what they say, and anything else that is not a
mapping is refused. The mapping's other fields are FlakySearch's:
pool, coverage_depth, threshold, epsilon, perturbations, draws
and directions. The space holds every parameter of a case: the attributes
drawn, the weather effects' quantiles, and where in its range the grip and
the visibility are (quantile(tire_friction_quantile),
quantile(visibility_quantile)), so two cases alike but for their grip are
two points, not one point whose outcome looks unsure. Only runs drawn in the
same space -- the same attributes, ranges and labels -- with an outcome are
learnt from (runs drawn before the grip and visibility were parameters, in
a space without them, are not); a sweep with no such
run draws all its cases with the sampler's own policy. Each case chosen for
its instability records the flakiness predicted for it, beside the one the
report assesses. Run the sweeps one after another, each reading the earlier
ones' outputs, and every sweep looks where the last left the outcome unsure.
Checking an ODD¶
uv run scenario-odd check my_package.odds:urban_odd # compile with Codon
uv run scenario-odd check path/to/urban.yaml # read the binding and OpenODD files
uv run scenario-odd show path/to/urban.yaml # attributes, buckets in and out
uv run scenario-odd list # ODDs known by name
check compiles a Python ODD with Codon, the same way a scenario is
compiled. It catches the following:
- a misspelt name;
- a condition of the wrong type (
between("0", 60)); - a probe that does not take a world.
The runner makes the same check before a run, under the typecheck config
key. OpenODD YAML is checked as it is read (see What is read).
Checking a planned route¶
Before a scenario runs, its ego route can be checked against the ODD on the Lanelet2 map alone, with no CARLA server:
uv run scenario-odd route path/to/urban.yaml intersection_passing/left_turn
uv run scenario-odd route urban 'intersection_passing/*' lane_change/left map=nishishinjuku
uv run scenario-odd route urban 'lane_change/*' --json
Each scenario config is composed as a run composes it. Name it as
scenario= does (a glob works too); arguments with an = are Hydra
overrides for all of them. The route is planned from the ego's spawn pose
(ego.spawn_lanelet_id, ego.spawn_s), through ego.waypoint_lanelet_ids,
to the goal (ego.goal_lanelet_id, ego.goal_s), as the shortest path in
the map's routing graph, leg by leg. A scenario that derives its goal in
setup() from scenario.expected_route_lanelet_ids, as
intersection_passing does, gets that goal here too (a configured goal
wins). A goal derived any other way is only known once the scenario runs:
that route is reported as not planned. So is an exported scenario on a map of
its own composed on another map (map=<id>_map left out), before any map is
read: its lanelet ids are its own map's
(Running an authored scenario).
The ODD's attributes are then read lanelet by lanelet along the route, and each lanelet gets one of three verdicts:
| Verdict | Meaning |
|---|---|
| inside | The ODD holds there, whatever happens at run time. |
| outside | The ODD fails there, whatever happens at run time. Reported with the lanelet ids and the modules that fail. |
| undecided | It depends on what only the run can tell (weather, traffic, speed), or on a value the map does not have. At run time a missing value counts as inside, so this is "inside only by assumption". |
The map decides an attribute when its probe has a lanelet form,
probe.on_lanelet(lanelet, lanelet_map, routing_graph). It returns the
value, None where the run would read none, or UNDECIDED where only the
run can tell (for example speed_limit_kph on a lanelet with no
speed_limit tag, where the run falls back to CARLA's limit). The built-in
probes listed above have one; a custom probe opts in by having an
on_lanelet attribute (a function attribute, or a method of a callable
object). An attribute whose probe has none is open: it could take any
value, and the ODD's own three-valued evaluation decides, exactly as it does
for buckets outside the ODD. An OpenODD attribute derived from others is
worked out from their values on the lanelet, and is open while one of them
is.
What the lanelet forms read, and where that differs from the run:
| Probe | On a lanelet | At run time |
|---|---|---|
in_junction |
The lanelet has a turn_direction tag (Autoware tags every lanelet in an intersection with one): lanelet_junction.is_junction_lanelet, which the sweeper's is_junction, its bindings and the route search ask too, so where a sweep places a scenario and what a run measures never disagree |
The same, on the ego's lanelet; without a Lanelet2 map, CARLA waypoint is_junction |
junction_type |
intersection where in_junction is true, else nothing |
The same, by in_junction |
lane_count |
The lanelet and every same-direction neighbour, walked left and right in the routing graph (left/right, else adjacentLeft/adjacentRight, so lanes behind a solid line count too); nothing in a junction |
The same, on the ego's lanelet; without a Lanelet2 map, the driving lanes of the same sign on the CARLA waypoint's OpenDRIVE road |
speed_limit_kph |
The speed_limit tag, else (no tag, or not a finite number above 0: nan, inf, 0, -30) undecided |
The tag, else (the same) CARLA's get_speed_limit() |
lanelet_location, lanelet_subtype, lanelet_speed_limit_kph |
The tag | The tag of the lanelet the ego is on |
A run with a Lanelet2 map reads the junction and the lane count off the ego's lanelet, as the route check does, so the two agree. A run without one falls back to CARLA's OpenDRIVE waypoints. The two maps are converted from each other but not always lane for lane (two lane-change neighbours on the Lanelet2 map can be the single lanes of two OpenDRIVE roads), so there the route check and the run can disagree.
The report also gives the expected coverage: the metres the route spends
in each bucket of each attribute (<undecided> and <missing> collect the
metres where the map cannot or does not say). The start lanelet counts from
the spawn pose, the goal lanelet up to the goal pose, and a lane change
splits the lanelets' length between the two sides. Over several scenarios
the metres are summed, and the report lists the buckets of each map-decided
attribute that no planned route reaches, leaving out buckets outside the
ODD (they are not targets). An attribute the map decides in general but left
undecided on some lanelet of a route (metres in <undecided>) is reported as
undetermined instead: the run may reach any of its buckets there, so none
is called unreached.
The exit status is 0 when no route leaves the ODD, 1 when one does, and 2 when a route could not be planned (no goal, no path, a lanelet the map does not have, no Lanelet2 map) or the ODD could not be read.
From Python:
from autoware_carla_scenario.odd import (
PlannedRoute, combine_route_coverage, plan_route_coverage,
)
from autoware_carla_scenario.coordinate import Lanelet2Pose
route = PlannedRoute(start=Lanelet2Pose(203, 25.0), goal=Lanelet2Pose(207, 10.0))
coverage = plan_route_coverage("urban", route, lanelet_map=lanelet_map)
coverage.leaves_odd, coverage.outside(), coverage.expected_m
combine_route_coverage("urban", [coverage, ...]).unreached
plan_route_coverage also takes a scenario (its spawn pose, waypoints and
goal) or a list of lanelet ids. Without lanelet_map it uses the map the
runner has loaded.
At start-up, once setup() has settled the goal, the runner plans the
route the same way and logs a warning when it leaves the ODD. It is a
warning only. It is skipped for an ODD with no modules, and anything that
keeps the route from being planned is logged at debug level; the run goes
on regardless.
What the report says¶
For each ODD the merged runs were measured against, scenario-coverage says
the following (Coverage):
- how long the runs were inside it, inside only because values were missing, and outside it;
- which modules ruled ticks out, and how often, and how often their verdict rested on missing values;
- which runs left it, and when (the first intervals);
- which attributes are monitored but not covered.
A bucket is outside the ODD when the ODD fails for every value in it,
whatever the other attributes are. It is reported, but it is not a coverage
target. An example is nonurban under location.is_in(["urban"]).
To decide this, the ODD's own evaluation runs with the bucket in place of the
attribute's value and every other attribute left open. Labels, inactive
modules, groups and module references therefore count exactly as they do on
a tick. A condition that ties attributes together (any_of across two of
them, or a junction and a speed) rules out combinations, never a single
bucket. A bucket only partly inside ([60, 90] under <= 60) stays a
target.