Software Architecture¶
This document describes the software architecture of the autoware-carla-scenario package, with a focus on the scenario execution system and the result viewer.
System Overview¶
The package consists of four major subsystems:
graph TB
subgraph CLI["CLI Entry Points"]
scenario["scenario command<br/>(Hydra)"]
viewer_cmd["viewer command<br/>(FastAPI + Uvicorn)"]
detect["detect-no-3d-model<br/>(argparse)"]
end
subgraph Execution["Scenario Execution System"]
queue["ScenarioQueue"]
runner["ScenarioRunner"]
base["BaseScenario"]
server["CarlaServerManager"]
end
subgraph Evaluation["Condition & Action System"]
conditions["Conditions"]
actions["Actions"]
end
subgraph Viewer["Result Viewer"]
app["FastAPI App"]
scan["Scanner"]
ui_runner["Runner (subprocess)"]
sweep_res["Sweep Resolver"]
end
subgraph Support["Support Modules"]
coord["Coordinate Transforms"]
entity["Entity Management"]
kin["Kinematics"]
sweeper["Lanelet Constraint Sweeper"]
recorder["Camera Recorder"]
end
scenario --> queue
viewer_cmd --> app
queue --> runner
queue --> server
runner --> base
runner --> conditions
runner --> actions
runner --> recorder
base --> entity
base --> coord
app --> scan
app --> ui_runner
app --> sweep_res
ui_runner -->|"uv run scenario<br/>(subprocess)"| scenario
Scenario Execution System¶
The scenario execution system is responsible for the full lifecycle of a CARLA scenario test: server management, map loading, actor spawning, tick-loop execution, condition evaluation, recording, and cleanup.
Architecture Diagram¶
graph TB
subgraph Entry["Entry Layer"]
main["main()<br/>examples/run.py"]
hydra["Hydra Config<br/>Compose API"]
end
subgraph Orchestration["Orchestration Layer"]
SQ["ScenarioQueue"]
SR["ScenarioRunner"]
CSM["CarlaServerManager"]
end
subgraph Scenario["Scenario Layer"]
BS["BaseScenario (ABC)"]
IP["IntersectionPassingScenario"]
LC["LaneChangeScenario"]
TLC["TrafficLightComplianceScenario"]
TS["TemporaryStopScenario"]
end
subgraph Traffic["Traffic Backend"]
tb["TrafficBackend<br/>(traffic_manager | none | third-party)"]
end
subgraph CARLA["CARLA Simulator"]
world["carla.World"]
tm["TrafficManager"]
rec["Native Recorder"]
end
main --> hydra
hydra --> SQ
SQ --> CSM
SQ --> SR
SR --> BS
BS --> IP & LC & TLC & TS
SR --> world
SR --> tb
tb --> tm
SR --> rec
Component Responsibilities¶
| Component | File | Responsibility |
|---|---|---|
CarlaServerManager |
server.py |
Start/stop/reuse the CARLA UE5 process. Manages lifecycle via process groups. |
ScenarioQueue |
scenario_queue.py |
Batch execution of multiple scenarios. Owns server and runner lifecycle. Context manager. |
ScenarioRunner |
scenario_runner.py |
Single-scenario execution: sync mode, tick loop, condition evaluation, recording, cleanup. |
BaseScenario |
scenario_base.py |
Abstract base. Subclasses implement setup() and is_done(). Registers conditions, actions, entities. |
TrafficBackend |
traffic/ |
Owns the vehicles the scenario did not author and answers their manoeuvre intents. traffic_manager by default; see Traffic Backends. |
Execution Lifecycle¶
The following sequence diagram shows the complete lifecycle of a single scenario execution:
sequenceDiagram
participant CLI as CLI (main)
participant SQ as ScenarioQueue
participant CSM as CarlaServerManager
participant SR as ScenarioRunner
participant BS as BaseScenario
participant EGO as Ego entity
participant W as carla.World
participant TB as TrafficBackend
participant TM as TrafficManager
CLI->>SQ: add(scenario)
CLI->>SQ: __enter__() / start()
SQ->>CSM: start()
Note over CSM: Reuse existing server<br/>or launch new process
SQ->>SR: new ScenarioRunner(server, ...)
SQ->>SR: load_map_by_name(), or<br/>load_map_by_overwriting_xodr()<br/>when map.overwrite_xodr is on
SQ->>SR: Initialize MapManager<br/>(Lanelet2 + OpenDRIVE)
CLI->>SQ: run_all()
loop For each scenario (each attempt, up to 1 + cooldown_max_retries)
Note over SQ: A dead server the queue owns is restarted first
opt The world is dirty (a run, or a failed attempt, left it)
SQ->>SR: reset_world(strict=True) (map reload, or soft reset)
Note over SQ: A reset that fails is this attempt's failure:<br/>retried (reset again), or the case errors
end
SQ->>SR: run_scenario(scenario, reset_world=False)
rect rgb(240, 248, 255)
Note over SR: Phase 1: Setup
SR->>W: Reload the runs' map if another client loaded a world
SR->>W: Destroy leftover dynamic actors
SR->>W: apply_settings(sync=True, dt=0.05)
SR->>TB: prepare(TrafficContext)
TB->>TM: set_synchronous_mode(True)
TB->>TM: set_random_device_seed(seed)
SR->>BS: set_client(client, tm_port)
SR->>BS: set_traffic_backend(backend)
SR->>BS: setup()
Note over BS: Spawn NPCs, register<br/>conditions & actions
SR->>W: Spawn ego vehicle
SR->>BS: register_fail_condition(EntityExistence)
SR->>W: start_recorder(path)
end
rect rgb(255, 248, 240)
Note over SR: Phase 2: Warm-up and start-up
loop STABILIZE_TICKS times (5 by default)
SR->>W: hold the vehicles still, tick()
end
SR->>BS: run_init (each init action once, at elapsed=0)
SR->>EGO: set_initial_speeds(...)
SR->>EGO: on_scenario_start(world)
SR->>EGO: _wait_for_ego (until the ego is ready)
Note over SR: An ego that asks to terminate here<br/>ends the run (no tick loop)
SR->>W: Release the vehicles' brakes
SR->>TB: start(world, skip_actor_ids)
TB->>W: set_autopilot(True) on the vehicles it owns
SR->>BS: set_initial_speed(ego actor)
SR->>BS: register_fail_condition(Timeout) (unless registered)
end
rect rgb(240, 255, 240)
Note over SR: Phase 3: Tick Loop
loop While not is_done(), until a result
SR->>BS: pre_tick_actions.tick(world, elapsed)
SR->>BS: pre_tick_callbacks(world)
SR->>W: tick()
SR->>EGO: on_tick(world, elapsed)
SR->>TB: tick(world, elapsed)
SR->>BS: post_tick_actions.tick(world, elapsed)
SR->>BS: post_tick_callbacks(world)
SR->>BS: Check pass_conditions
Note over SR: If any pass condition<br/>returns ScenarioResult → PASS
SR->>BS: Check fail_conditions (not MetricConditions)
Note over SR: If any fail condition<br/>returns ScenarioResult → FAIL
SR->>EGO: termination_requested?
Note over SR: If True → not a pass (termination)
end
Note over SR: is_done() with no result → PASS
SR->>BS: Judge MetricConditions on the recorded frames
end
rect rgb(255, 240, 240)
Note over SR: Phase 4: Cleanup
SR->>EGO: on_scenario_end(world)
SR->>BS: close_conditions()
SR->>W: Destroy ego
SR->>W: stop_recorder()
SR->>TB: close()
TB->>TM: shut_down()
SR->>SR: Write coverage, gate record and result JSON
end
SR-->>SQ: ScenarioResult
Note over SQ: The world stays dirty until the next attempt
end
CLI->>SQ: __exit__() / stop()
opt Server not stopped by the queue (given, or reused) and world dirty
SQ->>SR: restore_world() (best-effort reset, settings restored)
end
SQ->>CSM: stop() (if owned)
run_scenario called on its own (reset_world=True, the default) resets
the world at its end, best effort: a reset that fails is logged, not raised.
The queue instead passes reset_world=False and resets strictly before the
attempt that follows (reset_world(strict=True)), so that a world it could
not reset fails that attempt -- retried, or erroring the case -- rather than
being the world the next case runs on.
Tick Loop Detail¶
The tick loop runs at a fixed 20 Hz (0.05 s per tick) in CARLA synchronous mode. Each tick follows a strict evaluation order:
┌─────────────────────────────────────────────────────────────────┐
│ Single Tick Cycle │
├─────────────────────────────────────────────────────────────────┤
│ 0. is_done() check → True → exit (PASS if no result) │
│ 1. Pre-tick actions → action.tick(world, elapsed) │
│ 2. Pre-tick callbacks → callback(world) │
│ 3. world.tick() → advance simulation by 0.05 s │
│ 4. Trajectory log → record the new frame │
│ 5. Ego entity → ego.on_tick(world, elapsed) │
│ 6. Traffic backend → backend.tick(world, elapsed) │
│ 7. Post-tick actions → action.tick(world, elapsed) │
│ 8. Post-tick callbacks → callback(world) │
│ 9. Action states, NPC → logged (collisions, falls) │
│ contacts │
│ 10. Ego progress, metric → frames kept for MetricConditions │
│ frames │
│ 11. Coverage → sample TICK / condition items │
│ 12. Periodic logging → ego OpenDRIVE position (1/s) │
│ 13. Pass conditions → first satisfied → PASS & exit │
│ 14. Fail conditions → first triggered → FAIL & exit │
│ 15. Ego termination → requested → not a pass, & exit │
└─────────────────────────────────────────────────────────────────┘
After the loop: MetricConditions are judged on the recorded frames.
Evaluation semantics:
- Pass conditions are checked first. The first condition to return a non-
NoneScenarioResultterminates the loop with a pass. - Fail conditions are checked only if no pass condition was satisfied. The first triggered fail condition terminates the loop with a failure. A
MetricConditionis not checked in the loop: it is judged on the recorded frames once the run is over, and can still fail a run the loop passed. - An ego termination request (e.g. the driver policy signalled one) is checked after the conditions, so a pass or fail on the same tick still decides the outcome; on its own it is not a pass.
- If the loop exits via
is_done()returningTruewith no result, the scenario is treated as passed. Every path that reaches the tick loop records onecondition_statusesentry per registered condition; an ego that asks to terminate before it (its stack never became ready) records none, since no condition was ever checked.
The initialization phase¶
register_init registers what runs once, after the warm-up and before the ego
is asked to start: run_init performs each action a single time at
elapsed=0 and never ticks it again. That suits an action that is over once
commanded (a light frozen, a speed handed over, a source's initial vehicles
placed). An action that keeps acting after the tick it fired on, or whose run
waits on an until the world has to reach, says so with
BaseAction.needs_tick_loop -- by default, whether it reissues its command
(reissues_while_running) -- and register_init puts it on the pre-tick loop
instead, where it fires on the first tick: a trajectory or a stalled object, a
rate-limited, until-bounded or reissue=True SetSpeedAction, a lane change waiting to
settle, a walk that may turn back, a played or generated pedestrian motion, an
adversarial manoeuvre or NADE traffic, a construction zone that appears late
or is cleared away, and a
RoutingAction that moves its entity itself. Left in the phase, such an action
would sit in runningState for the whole run with nothing driving it. A
TrafficSourceAction or TrafficSinkAction stays in the phase on purpose:
registered there it acts once and stops, as does a construction zone that
stands from the start for the whole run, placed before the ego starts.
run_init performs the actions in registration order and settles each one
(complete, or under way for one that runs on) right after it fires, which also
gives back the control domains it took. Two init actions on the same domain of
one entity therefore both run, and the one registered last is what stands. The
case that comes up in practice is an ego with a goal and an init Set Goal card
for it: the ego's own goal is registered as it is spawned, before the
document's cards, so the ego is handed both and keeps the card's (the
validator warns about it).
The scenario clock¶
_ScenarioClock supplies the elapsed every action, condition, entity and
backend receives, and the elapsed_seconds on a ScenarioResult. It reads
world.get_snapshot().timestamp.elapsed_seconds — simulated time, and
nothing else.
The world advances by fixed_delta_seconds per tick and the loop steps it as
fast as the slowest client allows (see max_tick_rate_hz), so how much
simulated time fits into a second of real time is a property of the host. A
scenario whose durations came off the wall clock would fire its triggers in
different places on a fast machine and a slow one — the determinism that
synchronous mode exists to provide.
That holds for ScenarioRunner.timeout_seconds as much as for a condition an
author wrote. It is registered as a default TimeoutCondition on every
scenario that registers no timeout of its own, so a scenario given sixty
seconds is given sixty of its own however fast the host runs.
A scenario that registers a timeout of its own has said how long it runs, and the runner adds no default to cut it short: a red light whose drawn red phase is 13.4 s, given seven seconds after the green, runs for 20.4 s even when the run's default is twelve. Exactly these fail conditions count as its own:
- a bare
TimeoutCondition; - an
OrConditionwith a child that counts (an OR fires as soon as any child does); - a
StickyConditionwhose child counts (it hands on the child's failure as it comes; it latches only a pass).
A timeout under an AndCondition or a NotCondition does not count — it fires
only on some other state of the world, so it does not bound the run — and
neither does one under a PersistentCondition, which resets on every failure
its child reports and so never fires around a timeout.
The run's default — ScenarioRunner(timeout_seconds=...),
run_scenario(timeout_seconds=...), ScenarioQueue.add(timeout_seconds=...),
and with them a scenario.timeout_seconds= override — therefore does not apply
to a scenario with a timeout of its own. A hand-written scenario that builds its
timeout from cfg.timeout_seconds still follows the override, since that is
the same key; one that builds it from another (traffic_light_compliance with
timeout_after_green_seconds set) does not. A DeclarativeScenario always has
one: it registers its document's timeout_seconds (or the Hydra override of
it) itself, so a timeout node in the document is a second limit beside it,
and whichever fires first fails the run. scenario.timeout_seconds=null, how a
declarative scenario falls back to its document's timeout, leaves the run's
default at 60 s, as an absent key does; a value that is not a number (a string,
a bool, a list) is refused with a ValueError naming the key before the run
starts. A sweep keeps the wall-clock
sweep.job_timeout_seconds guard over every job either way.
Wall-clock protection against a stuck simulator lives outside the scenario clock, where it belongs:
| Guard | Where | Covers |
|---|---|---|
| CARLA client RPC timeout (60 s) | ScenarioRunner.__init__ |
world.tick() stops returning |
sweep.job_timeout_seconds (120 s) |
sweeper config | a whole job overrunning |
The clock starts after the ego is ready, so an entity that needs an autonomy stack to come up does not spend the scenario's timeout booting. Simulation time does advance during that wait, which is why the start is measured rather than assumed to be zero.
An entity that gives up waiting (its stack never became ready, so it sets
termination_requested) ends the run right there, before it begins: a failed
result of the driver category ("Ego entity requested session termination:
run_scenario(reset_world=True) (the default, for a direct call; a queue
passes False and resets before its next attempt) resets the world at the
end of a run, and also after a run that raises, before the error propagates:
best effort, a reset that fails is logged and the run's own error is the one
raised. A KeyboardInterrupt (Ctrl-C) propagates at once, with no reset.
ScenarioQueue: Batch Execution and Retry¶
ScenarioQueue wraps ScenarioRunner to support sequential execution of multiple scenarios:
graph LR
subgraph ScenarioQueue
direction TB
A[Scenario 1] --> B[Cooldown]
B --> C[Scenario 2]
C --> D[Cooldown]
D --> E[Scenario N]
end
subgraph Retry["Retry Logic"]
direction TB
R1["Attempt 1"] -->|fail| R2["Cooldown, reset_world()"]
R2 --> R3["Attempt 2"]
R3 -->|fail| R4["..."]
R4 --> R5["Attempt N"]
R5 -->|fail| R6["RuntimeError, or an errored result (continue_on_error)"]
end
Key features:
- Cooldown: Configurable wait time (
cooldown_seconds) between consecutive scenarios to allow CARLA server cleanup. - Retry: A scenario whose run raises is retried up to
cooldown_max_retriestimes before raising an error; one that runs and fails is not. The lanelet sweeper follows the same rule for its forked cases. A retry runs the same scenario object again, and so does a secondrun_all(): before eachsetup()the runner resets what an earlier attempt registered (BaseScenario._reset_registrations()) -- its conditions, actions, callbacks, entities, coverage and measures -- back to what the scenario's__init__registered, and resets each of those (BaseCondition.reset(),BaseAction.reset()) to the state it was built in, so a condition that latched in a failed attempt cannot decide the retry, aonceaction that fired there fires again, and a closedCollisionConditionattaches a sensor of its own. - World reset: The queue owns every reset between runs (#1941). It runs each scenario with
run_scenario(reset_world=False)and keeps a flag of whether the world holds what a run left; right before the next attempt -- the next scenario's first, a retry, or the first of a laterrun_all()(the SLURM worker callsrun_all()once per case) -- it resets the world (ScenarioRunner.reset_world(strict=True)), so the weather, frozen traffic lights and light states one run set do not carry into the next. Between scenarios the reset comes first and the cooldown after it, so the server settles after the reload before the next scenario connects; a retry waits first (its recovery wait), then resets. The reset is part of that attempt, and strict: a map load that fails (a timed-outload_world, a server that died in it) is logged and raised rather than leaving the world as the last run left it (#2243), and the next reset loads the map. One that raises fails that attempt, which is retried (and resets again) or, withcontinue_on_error, errors that scenario, and never changes the verdict of the scenario that ran before it, nor ends the batch. An attempt that failed before its reset (in a retry's wait, or a server restart) leaves the flag as it was. After the last scenario the world is left as it is, for the nextrun_all()to reset, unless the server is restarted (a new world) in between. When the queue stops (stop(),__exit__, the end of anas_fixture()session) with the world dirty, a server it does not stop -- one it was given, or one its own manager found already running and reused -- would be left synchronous at 20 Hz with nothing ticking it, and with the last run's actors in it, frozen for every other client (#2067). Sostop()first restores it (ScenarioRunner.restore_world()): it resets the world once more (reset_world(), which destroys the runs' actors) and applies the world settings the runner's first run found (asynchronous, no fixed time step, for a freshly loaded map; a queue that loads no map puts back whatever its first run found, synchronous settings included). It leaves the TrafficManager alone: each run's traffic backend already shut its own down at the end of the run, and asking the client for one would start a new one, or take over another client's on that port. A server that does not answer a quick ping (CarlaServerManager.is_alive(), a few seconds) is not restored, with a warning: each call would wait for the client's 60 s timeout, or abort the process on arm64, before the launcher is stopped. Each step is best effort: a failure is logged, the next step still runs, andstop()goes on to close the Autoware launcher and the backends without raising. A server the queue stops gets none of this. The single-scenario run (run_scenario_with_queue()), batch mode, the SLURM worker and the sweeper's jobs (which run batch mode) all stop their queue withwith queue:, so each restores the world this way. Arun_all()that raises (a scenario failing every attempt withoutcontinue_on_error, or Ctrl-C) leaves the world as it was too: the nextrun_all()resets it before its first scenario. Withrun_all(continue_on_error=True), which batch mode uses, a scenario that fails every attempt is recorded instead as a failed result naming the error (errored_indices()tells it from a scenario that ran and failed), and the queue goes on with the next one, whose first attempt resets the world. - Server restart: A CARLA server that dies mid-batch cannot recover by waiting, so before every attempt -- each case's first, and every retry -- a queue that owns its server checks it (
CarlaServerManager.is_alive()) and, when it is gone, logs it, stops it and redoes the server part ofstart(): the server is started again (on the same port and arguments, so a SLURM task keeps its port slot), a newScenarioRunnerconnects to it, and the map is loaded and theMapManagerbuilt again. The backends and the Autoware launcher are not prepared again: neither talks to the CARLA server. A restart that raises is the failure of the attempt it came before, retried or errored like any other, and the next attempt restarts again. A server the queue was given (ScenarioQueue(server=...)), or one its own manager found already running and reused (CarlaServerManager.reused), is never restarted, and a server that is alive is never touched. The SLURM worker, which callsrun_all()once per case on one queue, gets the same restart before each case. - Server ownership: Accepts an external
CarlaServerManageror creates and owns one internally. Astart()(or__enter__) that fails -- the runner cannot connect, the map name is unknown, the Lanelet2 map does not load -- callsstop()before the error propagates, so the server the queue owns is stopped and the Autoware launcher closed even though__exit__does not run. - Server start:
CarlaServerManager.start()never takes another process's server for the one it launched (#2233). Before launching on a local host it tries a plain TCP connection tohost:port: withreuse_if_running=Falsea port that accepts one is refused at once ("in use by another process"; stop that process, choose anotherserver.port, or setreuse_if_running=Trueto use the server there), and withreuse_if_running=Truea port that accepts a connection but did not answer the first ping as CARLA (a busy server can miss its 2 s deadline) is pinged again for up toCarlaServerManager.REUSE_PING_S(30 s) and reused once it answers, or refused -- never launched over, since a second server could not bind the port; a port that stops accepting connections during that wait (a server still shutting down, as after a queue's restart stopped it) is launched on. While a launched server starts, its own process is checked before each ping and after the one that answers: once it has exited,start()fails with its exit code and where CARLA's own log is (<launcher directory>/CarlaUnreal/Saved/Logs), even when something answers on the port. - Server lifetime: A CARLA that
CarlaServerManagerlaunched does not outlive the process that launched it.stop()(orwith, oratexit) stops it on a normal exit; a SIGTERM toscenarioraisesKeyboardInterrupt, so thewithblocks unwind as on Ctrl-C, and it exits 143; and CARLA is launched under a small watchdog that leads its process group and, once that process is gone (killed with SIGKILL, say, as the lanelet sweeper does a job pastsweep.job_timeout_seconds), sends the group SIGTERM and, 10 s later, SIGKILL; the sweeper then waits until that CARLA has exited before its next attempt. The watchdog polls its parent process rather than usingPR_SET_PDEATHSIG, which follows the thread that started it. A server that was already running and was reused is never stopped. - pytest integration:
as_fixture()generates a session-scoped pytest fixture with automatic skip when the queue fails tostart()-- among other reasons, when no server answers at its host and there is none to launch. Whether a CARLA is installed is not asked first: a remote server, or one already running, needs none. It runs the scenarios withrun_all(continue_on_error=True), so a scenario that raises on every attempt fails its own tests rather than skipping the session: its result is a failed one naming the error, and aCarlaScenarioFixturefor it fails each test that uses it.
Recordings of a run¶
A run leaves two recordings, neither of which renders anything:
- The CARLA recording (
{stem}.log): CARLA's native recorder captures every actor's state from the moment the ego exists, warm-up included. CARLA's replayer (client.replay_file()) plays it back on a server running the same map. - The trajectory log (
{stem}.trajectory.jsonl, Trajectory Log): written by the runner itself after every tick of the scenario, in-process -- every vehicle and pedestrian, the lights, the map the run was on, the ego's debug markers, each condition's state and the verdict. A Rerun recording made from it with theautoware_carla_scenario_rerunpackage is how a run is seen (Seeing a run): withrerun.enabled=truethe runner hands each closed log to aRerunConverter(rerun.py), which converts it on a background thread while the next case runs, and the run waits for it at the end.
Releases before 15.0.0 also rendered a video from the CARLA recording after each run ({stem}.mp4, a replay with an RGB camera and ffmpeg); that is gone, and so is the ffmpeg it needed.
Debug markers¶
The ego entity can show what the world does not: EgoVehicle.debug_markers() returns what it is following on the current tick (DebugMarkers, polylines in CARLA world coordinates, from autoware_carla_scenario.debug_markers). The base class shows nothing; the entities that follow a trajectory show it, as a green line with a dot at each point:
| Entity | What it shows |
|---|---|
CarlaDriverEntity |
The driver policy's current plan |
Ros2EgoEntity |
The trajectory it is tracking, while a trajectory command source drives |
After every tick of the scenario the runner asks the ego for its markers and writes them into the trajectory log when they change (a markers record). An entity hands out the same DebugMarkers object until its plan changes (plan_markers() builds one per plan), so the log holds one per plan rather than one per tick. A viewer of the run draws them beside the path the ego drove, which the log's ticks give.
An entity of your own shows more by overriding debug_markers():
from autoware_carla_scenario.debug_markers import DebugMarkers, DebugPolyline
class MyEgo(EgoVehicle):
def debug_markers(self) -> DebugMarkers:
markers = DebugMarkers()
markers.add_polyline(
DebugPolyline(self._path_points, color=(255, 0, 0), point_size=0.1)
)
return markers
Condition and Action System¶
Condition Architecture¶
Conditions evaluate scenario pass/fail criteria. All conditions inherit from BaseCondition:
classDiagram
class BaseCondition {
<<abstract>>
+label: str
+check(world, elapsed) ScenarioResult | None
+get_details() dict
+to_summary_dict() dict
+close()
}
class ScenarioResult {
+passed: bool
+message: str
+elapsed_seconds: float
+condition_statuses: list
+failure: Failure
}
class TimeoutCondition
class ElapsedTimeCondition
class CollisionCondition
class EntityExistenceCondition
class OffRoadFractionCondition
class PlaceClearCondition
class EntityCountCondition
class TrafficSignalCondition
class TrafficSignalControllerCondition
class RouteProgressCondition
class ActionStateCondition
class TrajectoryTimeCondition
class LaneChangeSettledCondition
class MetricCondition
class AdversityFeasibleCondition
class AdversityActiveCondition
class CompositionCondition {
<<abstract>>
+_check(world, elapsed) ScenarioResult | None
}
class EntityLanePositionCondition
class EntityDistanceCondition
class EntityPositionDistanceCondition
class EntityAreaDistanceCondition
class TimeToCollisionCondition
class TimeHeadwayCondition
class RelativeSpeedCondition
class SpeedCondition
class AccelerationCondition
class StandstillCondition
class TemporaryStopCondition
class TurnIndicatorCondition
class WaypointCondition
class EntityLaneOfCondition
class AndCondition
class OrCondition
class NotCondition
class StickyCondition
class PersistentCondition
class AlwaysTrueCondition
BaseCondition <|-- TimeoutCondition
BaseCondition <|-- ElapsedTimeCondition
BaseCondition <|-- CollisionCondition
BaseCondition <|-- EntityExistenceCondition
BaseCondition <|-- OffRoadFractionCondition
BaseCondition <|-- PlaceClearCondition
BaseCondition <|-- EntityCountCondition
BaseCondition <|-- TrafficSignalCondition
BaseCondition <|-- TrafficSignalControllerCondition
BaseCondition <|-- RouteProgressCondition
BaseCondition <|-- ActionStateCondition
BaseCondition <|-- TrajectoryTimeCondition
BaseCondition <|-- LaneChangeSettledCondition
BaseCondition <|-- MetricCondition
BaseCondition <|-- AdversityFeasibleCondition
BaseCondition <|-- AdversityActiveCondition
BaseCondition <|-- CompositionCondition
CompositionCondition <|-- EntityLanePositionCondition
CompositionCondition <|-- EntityDistanceCondition
CompositionCondition <|-- EntityPositionDistanceCondition
CompositionCondition <|-- EntityAreaDistanceCondition
CompositionCondition <|-- TimeToCollisionCondition
CompositionCondition <|-- TimeHeadwayCondition
CompositionCondition <|-- RelativeSpeedCondition
CompositionCondition <|-- SpeedCondition
CompositionCondition <|-- AccelerationCondition
CompositionCondition <|-- StandstillCondition
CompositionCondition <|-- TemporaryStopCondition
CompositionCondition <|-- TurnIndicatorCondition
CompositionCondition <|-- WaypointCondition
EntityLanePositionCondition <|-- EntityLaneOfCondition
BaseCondition <|-- AndCondition
BaseCondition <|-- OrCondition
BaseCondition <|-- NotCondition
BaseCondition <|-- StickyCondition
BaseCondition <|-- PersistentCondition
BaseCondition <|-- AlwaysTrueCondition
BaseCondition --> ScenarioResult : returns
Condition categories:
| Category | Conditions | Description |
|---|---|---|
| Temporal | TimeoutCondition, ElapsedTimeCondition |
Time-based triggers |
| Safety | CollisionCondition, EntityExistenceCondition, OffRoadFractionCondition, PlaceClearCondition, EntityCountCondition |
Collision detection — with anything, or with a named entity or class of object — actor alive checks, leaving the driving lanes by more than a tolerance, a place clear of road users, and how many road users of a kind are where |
| Position | EntityLanePositionCondition, EntityLaneOfCondition, WaypointCondition, RouteProgressCondition |
Road/lane position (a named lane, or the lane of a position), no road ahead within a distance (WaypointCheckType.IS_EMPTY), distance along the route |
| Relative | EntityDistanceCondition, EntityPositionDistanceCondition, EntityAreaDistanceCondition, TimeToCollisionCondition, TimeHeadwayCondition, RelativeSpeedCondition |
Gap to another entity, to a place on the map, or from an entity's body to an area of the map chosen by its ground (along the road with measure: lane; signed there for ahead of the subject, the way it faces, positive, behind negative), time to collision, following headway and speed difference |
| Motion | SpeedCondition, AccelerationCondition, StandstillCondition, TemporaryStopCondition, TurnIndicatorCondition |
Speed and acceleration thresholds, standstill detection, stop-and-go, turn indicators |
| Traffic | TrafficSignalCondition, TrafficSignalControllerCondition |
One light's state, or a whole junction's phase |
| Actions | ActionStateCondition, TrajectoryTimeCondition, LaneChangeSettledCondition |
Another action's lifecycle state, a trajectory's clock, a lane change settled |
| Adversity | AdversityFeasibleCondition, AdversityActiveCondition |
Whether an adversity can be carried out here, and whether one is in progress |
| Metric | MetricCondition |
A metric NG for longer than allowed |
| Composition | AndCondition, OrCondition, NotCondition |
Logical combinators |
| Stateful | StickyCondition, PersistentCondition |
Latch once satisfied / persist across ticks (a persistent condition counts only passing ticks, so it never fires around a condition that only reports a failure -- the collision, off-road, existence and timeout ones -- and warns when built around one: Usage) |
| Utility | AlwaysTrueCondition |
Unconditional trigger (default for actions) |
A collision with map geometry is a collision. CARLA gives no actor for the other side of a collision with a wall, a kerb or another part of the map that is not an actor (
CollisionEvent.other_actorisNone). An untargetedCollisionCondition(notarget,target_typeANY) fires on it, and its details sayother_actor_type: "not an actor". A targeted one ignores it, as it ignores any collision that is not the one named, and keeps waiting for that one (#277).Distances carry a coordinate system.
EntityDistanceCondition,TimeHeadwayConditionandTimeToCollisionConditioneach take acoordinate_system, which is OpenSCENARIO'scoordinateSystemattribute:
ENTITY(the default, so existing documents keep their meaning) is the straight-line separation in the world frame: on a curve, the chord. The directional measures project it onto the subject's axes: aLONGITUDINALorLATERALdistance along or across its heading, a headway along its direction of travel. The chord and the projection along the subject equal the along-lane gap on a straight road and under-read on a curve: for a leader 20 m ahead along the lane, the chord (2R·sin(L/2R), aEUCLIDEANdistance or a TTC's range) reads 19.9 / 19.5 / 18.6 m at R = 50 / 25 / 15 m, the projection (R·sin(L/R), a headway or aLONGITUDINALdistance) 19.5 / 17.9 / 14.6 m. Past a half turn (an arc of π·R or more) the projection changes sign, as does a TTC's closing speed, taken along the chord: a headway and a TTC both stop firing for a leader that is ahead along the lane. ALONGITUDINALdistance takes the magnitude and does not.LANEprojects both entities onto the OpenDRIVE road they share and compares theirs, which is whatscenario_simulator_v2measures. For TTC the closing speed is resolved onto the road too, so a vehicle's cornering is not counted as approach.
LANEfollows the chain of connected roads between the two, which it has to: an OpenDRIVE map is cut into short roads — a median of 33 m on the converter's own fixture — so a leader at an ordinary following distance is usually on the next road, not the one behind it. Measuring within one road would answer "no measurement" to most following scenarios.A junction is measured too. Its connecting roads are ordinary roads carrying
junction="<id>", so the chain reaches them like any other, and two entities on one connecting road need no walk at all — which is the measurement an intersection scenario is usually about. What a junction cannot settle is which way out a vehicle will take, so that ambiguity is reported rather than hidden: where several chains reach the target the shortest answers, by distance rather than by which link the map happens to list first, and every measurement carries whether it ran through a junction. A caller that must not have a turn chosen for it passesacross_junctions=Falseand gets no measurement instead.Nothing falls back to the straight line: silently swapping one measure for another makes a scenario pass for the wrong reason. Bounded at 200 m, which covers a motorway-speed TTC over the seconds a threshold is set at. The measurement itself lives in
autoware_carla_scenario.coordinate.lane_distance.
check() contract:
- Returns
ScenarioResultwhen the condition is triggered (satisfied or violated). - Returns
Nonewhen not yet triggered. __init_subclass__auto-wrapscheck()to track the latest result for UI display.
Action Architecture¶
Actions execute side effects when their trigger condition is met:
classDiagram
class BaseAction {
<<abstract>>
+label: str
+timing: TickTiming
+once: bool
+state: ActionState
+reissues_while_running: bool
+execute(world) void
+tick(world, elapsed) void
}
class BaseCondition {
<<abstract>>
}
class TrafficSignalAction
class TrafficSignalControllerAction
class EnvironmentAction
class TurnAction
class WalkStraightAction
class PlayMotionAction
class SmartPedestrianControllerAction
class LaneChangeAction
class RoutingAction
class SetSpeedAction
BaseAction <|-- TrafficSignalAction
BaseAction <|-- TrafficSignalControllerAction
BaseAction <|-- EnvironmentAction
BaseAction <|-- TurnAction
BaseAction <|-- WalkStraightAction
BaseAction <|-- PlayMotionAction
BaseAction <|-- SmartPedestrianControllerAction
BaseAction <|-- LaneChangeAction
BaseAction <|-- RoutingAction
BaseAction <|-- SetSpeedAction
BaseAction --> BaseCondition : trigger condition
Action lifecycle:
Each tick the runner calls action.tick(world, elapsed), which walks the action
through the OpenSCENARIO storyboard element states held in ActionState
(standbyState -> startTransition -> runningState -> endTransition ->
completeState).
- Standby. The trigger
BaseConditionis evaluated viacondition.check(world, elapsed). A non-Noneresult fires the action:execute(world)runs and the action entersstartTransition. - Running. The trigger cannot fire, so one run cannot begin on top
of another. A repeating action (
once=False) still checks it every tick and drops the result, so a stateful trigger (PersistentCondition,StickyCondition) sees every tick, as the children ofAnd/Ordo. Two independent questions are then asked every tick, in this order:
Does execute repeat? — reissues_while_running. This is a fact about
the command, not about the manoeuvre: a TrafficManager target persists
until it is changed, while a command that walks a target towards a goal has
to be re-sent on every tick to move at all. Because that depends on what
drives the entity, it is resolved in the order the knowledge is available:
| Source | Who knows |
|---|---|
reissue= constructor argument |
whoever knows the backend — the injection point |
_reissues_by_default() override |
the action, from its own configuration or by asking its entity |
REISSUES_BY_DEFAULT class attribute |
the action's class, when it has one answer |
When does the run end? — until, a BaseCondition. This is the same
kind of object the trigger is written in, so the two ends of a run read
together. No until means there was nothing to wait for, and the run is
over on the tick after the one it was commanded on. until is reset
(BaseCondition.reset()) as each run of a repeating action after its
first begins, before execute, so a run never ends on a latch or a hold
timed in an earlier run. (The runner resets it before the first run.)
Keeping the two apart is what lets both shapes exist. LaneChangeAction
hands work to the simulator and watches for it to land — the traffic
backend's lane change (TrafficManager force_lane_change, SUMO
changeLane) returns long before the vehicle is in the next lane, and must
not be re-sent
— so it supplies a LaneChangeSettledCondition as its until and does not
repeat. A rate-limited speed change says both.
3. End. endTransition is held for one tick so a condition can watch for
it. Then once decides: True (default) means completeState and no
further evaluation; False returns the action to standby to be triggered
again.
The exception: an instantaneous action. runningState and
endTransition are only seen on an action that runs on -- one with an
until, or one that reissues its command. An action with neither is over
the moment it is commanded: on the tick after startTransition it goes
straight to completeState (standbyState with once=False, from which it
may fire again on that same tick), never showing runningState or
endTransition. Pausing on a transition it passed straight through would
only make every instantaneous action a tick slower to repeat.
| Action | startTransition |
runningState |
endTransition |
|---|---|---|---|
No until, no reissue |
one tick | never | never |
No until, reissues |
one tick | never (the run is one reissued tick) | one tick |
until |
one tick | every tick until has not yet held (none if it holds on the tick after firing) |
one tick |
Stopped by an overrides=True action |
as above | as above, until it is stopped | never: straight to completeState, even with once=False (stopped later in the tick it fired, its startTransition is not recorded either) |
So a condition waiting on runningState or endTransition of an
instantaneous action never holds -- wait on completeState (or
startTransition) instead. A repeating instantaneous action whose trigger
still holds fires again on the very tick it would have gone back to
standbyState, so it stays in startTransition tick after tick and
standbyState is not seen until its trigger stops holding. An action
fired in the init phase never holds endTransition either: it is settled
there at once (_settle_start) -- to completeState (standbyState with
once=False) if it does not reissue its command and either has no until
or its until already holds at elapsed=0. One that reissues, or whose
until does not yet hold, is left in runningState, and since the init
phase never ticks it again it stays there for good, unless it was put on
the tick loop instead: register_init does that for every action whose
needs_tick_loop is true. An action can opt out, as the background-traffic
source and sink (TrafficSourceAction, TrafficSinkAction) do (a
ConstructionZoneAction with a later start or an end time does not): one
registered for initialization reissues by default
(REISSUES_BY_DEFAULT = True), so it places its vehicles (or clears its
lanelets) once and holds runningState for the rest of the run, even with
an until that already holds.
until, reissues_while_running and once are all independent — until says
when this run ends, reissues_while_running says whether the command repeats
while it does, and once says whether another run may begin.
The adversarial actions (TeraSim's adversities, one entity each), NADE
(AdversarialTrafficAction, which picks which agent carries out which
adversity, with its importance-sampling weight) and the scheduled stalled
objects build on this lifecycle: see
Adversarial Actions and NADE.
ActionStateCondition observes these states, which is what lets one actor react
to another finishing a manoeuvre rather than to the command having gone out
(done only reports the latter).
Control domains:
An action that commands an entity takes control of one or more of its
control domains, OpenSCENARIO's four (ControlDomain, in
autoware_carla_scenario.control_domain): longitudinal (how fast it goes),
lateral (where across it goes), lighting and animation. Longitudinal and
lateral together are its movement (MOVEMENT).
| Action | Domains of its entity |
|---|---|
SetSpeedAction |
longitudinal |
LaneChangeAction, TurnAction |
lateral |
TurnIndicatorAction |
lighting |
EmergencyAction |
lighting |
WalkStraightAction |
longitudinal, lateral |
RoutingAction, follow / position |
longitudinal, lateral |
RoutingAction, driver |
lateral; longitudinal too with a speed, a timed waypoint, stop_at_goal, or for a pedestrian (which it walks) |
| The adversarial actions | longitudinal, lateral |
AdversarialTrafficAction (NADE) |
longitudinal, lateral of each road user it drives, from when its path starts until it is handed back |
PlayMotionAction |
animation; longitudinal and lateral too with upper_body_only=False |
SmartPedestrianControllerAction |
longitudinal, lateral and animation |
The domains are taken as the action fires, before execute runs. An action
that runs on (it has an until, or reissues its command) holds them until its
run ends (endTransition, or close()); one that is over the moment it is
commanded holds them for that tick alone. An action that fires to command a
domain another action holds raises ControlConflictError, naming both, and
commands nothing: the run ends in error. So two speed commands to one vehicle
in the same tick, or a SetSpeedAction while a WalkStraightAction that turns
back is still watching, are errors, while a LaneChangeAction beside a running
SetSpeedAction is not.
Whether that is an error is the newer action's to say, one action at a time. An
action built with overrides=True (in the Scenario Editor, its card's
Override a running action) takes the domain over instead, as OpenSCENARIO has
every action do (Actions at runtime): the action holding it is stopped -- it
goes to completeState, OpenSCENARIO's stop transition -- and gives back every
domain it held, and any entity it took from its driver: a follow or
position RoutingAction brakes its vehicle (or stands its walker still) and
hands the vehicle back to its traffic backend, an adversity hands its road user
back, a routed walk is stopped. NADE is stopped only on the road user taken:
that one is handed back and NADE runs on; it never picks a road user another
action holds. A follow or position run that its until condition ends
before the end of its trajectory gives its entity back the same way. The
default is False: actions fighting over an entity say two things at once, and
which one CARLA would end up doing is an accident of the tick order.
A post-tick action and the pre-tick action of the next loop are handed the same time and act on the same simulation step, so they share a tick. The holds are process-wide, like the entity registry, and the runner forgets them as each scenario is set up.
Tick timing:
Actions can be registered as pre-tick (before world.tick()) or post-tick (after world.tick()) via BaseScenario.register_pre_tick() / register_post_tick().
Traffic signals: one light, or a junction's cycle¶
Two pairs of primitives, answering two different questions.
TrafficSignalAction / TrafficSignalCondition set and read one light.
That is enough to arrange a specific state and no more: a junction assembled a
light at a time can be left with two conflicting approaches green, a state no
real road reaches, and an amber has no name at all — it is just a colour one
light happens to be showing.
TrafficSignalControllerAction / TrafficSignalControllerCondition work in
phases. A phase names every signal its controller drives and how long it
holds, so showing one puts the whole junction into a known state at once, and
waiting for one is a statement about the junction rather than about a colour.
The one-light pair and the controllers find a Lanelet2 light's CARLA lights alike,
coordinate.find_traffic_lights_for_lanelet2_id: through the OpenDRIVE
controller named after it, or, on a map where none is (a roadgen map), by the
CARLA light whose stop waypoint lies on its stop line. A light an action sets
is therefore the light a condition reads and a controller's phases drive.
A junction is declared in two halves, split along what changes between scenarios and what does not.
map.signal_groups is the road. Which signals change as one, and which
movements cross each other, are facts about the junction: every scenario that
runs here inherits the same ones. Naming a group is also the only way a phase
table stays writable — one approach of three lanes is three regulatory
elements, and nishishinjuku declares 164 of them.
map.traffic_signal_controllers is this scenario's timing. A test of an ego
creeping into a long amber and a test of the same crossing on a short cycle are
two scenarios on one road, so the durations are the scenario's to choose.
OpenSCENARIO reads the same way: RoadNetwork/TrafficSignals holds the
controllers, and RoadNetwork is a section of the scenario file, beside the
LogicFile that names the map rather than inside it —
scenario_simulator_v2 builds its TrafficSignalController from that scenario
XML too.
map:
signal_groups: # the road: no colours, no durations
- name: ns
lanelet2_regulatory_element_ids: [1001, 1003]
conflicts_with: [ew] # read symmetrically
- name: ew
lanelet2_regulatory_element_ids: [1002]
traffic_signal_controllers: # this scenario's timing
- name: crossing
phases:
- name: ns_green
duration_seconds: 20.0
states:
- {group: ns, state: green}
- {group: ew, state: red}
- name: ns_amber
duration_seconds: 3.0
states:
- {group: ns, state: yellow}
- {group: ew, state: red}
- name: all_red
duration_seconds: 1.0
states:
- {group: ns, state: red}
- {group: ew, state: red}
- name: next_junction
reference: crossing # a green wave along a corridor is declared,
delay_seconds: 8.0 # not timed by a chain of actions
phases: [...]
A state may name a lanelet2_regulatory_element_id instead of a group, which
is the escape hatch for a single light — including the case below, where the
junction's impossible state is the thing under test.
autoware_carla_scenario.signals turns those declarations into running
SignalController objects at scenario setup. Each is registered as a
pre-tick callback and cycles on its own: a phase holds for its
duration_seconds and then hands over to the next, looping. No runner change
was needed — durations are differences, so the controller reads the world's own
simulated elapsed seconds directly.
The action jumps to a named phase and the cycle carries on from there, so a junction forced green does not thereby stay green; holding it is a separate decision, made by declaring a phase long enough to hold. The condition reads the phase off the controller, not off the lights: the controller is what decided them, so a junction that some other actor happened to set to the same colours does not read as that phase.
A reference offsets one controller's start from another's, which is how a
progressive system — a green wave — is written. The offsets are declared;
nothing has to time them.
Groups are what make the original problem checkable. Two conflicting
approaches green is the state a junction assembled one light at a time can
reach and a real road cannot; while a phase was a list of regulatory element
ids, nothing could tell. Now conflicts_with says which movements cross, so the
validator rejects any phase that shows green or amber to both — amber
included, because a junction showing amber to one movement still has vehicles in
it, which is why a real one puts an all-red clearance between them. A scenario
whose subject is a junction in that state names the regulatory elements
directly and is left alone: naming a group is how an author invokes the map's
statement about that movement, this rule included.
The validator also rejects a delay with no reference, a reference to a controller the document does not declare, a loop of them, duplicate controller, phase or group names, a signal in two groups, a group that drives nothing, a conflict naming an undeclared group, a phase state naming both a group and an element or neither, an unknown state, and any action or condition naming a controller or phase that does not exist — all while the document is being edited, because a misspelt phase name is invisible at runtime: the junction cycles normally and the scenario just waits forever.
Signal group, controller and phase names are read without surrounding
whitespace, wherever they are written: a group's name and conflicts_with, a
phase state's group, a controller's name and reference, a phase's name,
and an action's or condition's controller and signal_phase. A hand-written
"ns " is the group ns, so a name the validator accepts is the name the
running controller is found by.
Result Viewer¶
The result viewer is a web application for browsing, inspecting, and triggering scenario test results.
Architecture Diagram¶
graph TB
subgraph Browser["Browser"]
HTML["HTML Pages<br/>(Jinja2 templates)"]
SSE["SSE Client<br/>(EventSource)"]
JS["JavaScript<br/>(htmx + vanilla)"]
end
subgraph FastAPI["FastAPI Application (app.py)"]
direction TB
pages["Page Routes<br/>GET /, /session/..., /scenario/..."]
api["API Routes<br/>POST /api/run, /api/refresh<br/>GET /api/scenarios, /api/run/progress"]
static["Static Files<br/>CSS, JavaScript"]
end
subgraph Services["Service Layer"]
scanner_svc["Scanner<br/>(scanner.py)"]
runner_svc["Runner<br/>(runner.py)"]
sweep_svc["Sweep Resolver<br/>(sweep_resolver.py)"]
end
subgraph Storage["File System"]
outputs["outputs/<br/>YYYY-MM-DD/HH-MM-SS/"]
multirun["multirun/<br/>YYYY-MM-DD/HH-MM-SS/{0,1,2...}/"]
conf["conf/scenario/<br/>*.yaml"]
end
subgraph External["External Process (uv run scenario)"]
subprocess["subprocess.run()"]
ext_main["main()<br/>(examples/run.py)"]
ext_queue["ScenarioQueue"]
ext_runner["ScenarioRunner"]
ext_carla["CARLA Simulator"]
end
HTML --> pages
JS --> api
SSE --> api
pages --> scanner_svc
api --> scanner_svc
api --> runner_svc
api --> sweep_svc
scanner_svc --> outputs
scanner_svc --> multirun
scanner_svc --> conf
runner_svc --> subprocess
subprocess --> ext_main
ext_main --> ext_queue
ext_queue --> ext_runner
ext_runner --> ext_carla
sweep_svc --> conf
ext_runner -->|"*_result.json<br/>*.trajectory.jsonl"| outputs
ext_runner -->|"*_result.json<br/>*.trajectory.jsonl"| multirun
Component Responsibilities¶
| Component | File | Responsibility |
|---|---|---|
app.py |
ui/app.py |
FastAPI routes, template rendering, SSE streaming |
scanner.py |
ui/scanner.py |
Scan outputs/ and multirun/ directories for result JSON files. Build session lists and condition trees. |
runner.py |
ui/runner.py |
Execute uv run scenario as subprocesses in a background thread. Track progress with thread-safe global state. |
sweep_resolver.py |
ui/sweep_resolver.py |
Expand a scenario's sweep into concrete per-job override lists without launching CARLA, through the CLI's sweeper.expand.expand_config (constraints, route searches, ODD draws); a failure, or a scenario whose sweep expands to no case, is reported to the Run panel and nothing runs. The jobs run with the Run panel's extra overrides. |
models.py |
ui/models.py |
Pydantic data models for the viewer UI. |
cases.py |
ui/cases.py |
Which files are each case's: the stem its result, trajectory log, coverage and ODD archive share (a SLURM batch's manifest, or matched against batch_results.json). Its session_dir is the containment check every route and scanner function builds a session's directory with (resolved, it must stay under multirun/ or outputs/; else 404). |
results.py |
ui/results.py |
A session's cases with their ODD case, coverage and metrics off the trajectory log; the cases in a cell of openodd's coverage view; sessions compared, cases matched by what they ran. |
slurm_launch.py |
ui/slurm_launch.py |
Submit a batch through SLURM (launcher=slurm --no-wait) and follow it from its directory and squeue; resubmit, collect, cancel with scenario-slurm. |
slurm_options.py |
ui/slurm_options.py |
The launcher.* keys the run panel sets: what each may hold, the Hydra overrides they become (quoted as Hydra reads them back) and the scenario ... --no-wait command line. Checked by the library check (docs/typecheck.md). |
recordings.py |
ui/recordings.py |
A case's Rerun recording, made from its trajectory log on demand in a worker thread (the optional autoware_carla_scenario_rerun), and where the page loads Rerun's web viewer from (a copy it serves, or the CDN). |
Data Models¶
classDiagram
class SessionSummary {
+date: str
+time: str
+session_type: str
+scenario_name: str
+passed_count: int
+total_count: int
+pending_count: int
}
class SessionItem {
+index: int
+scenario_name: str
+passed: bool | None
+elapsed_seconds: float | None
+overrides: list[str]
+message: str
}
class ScenarioResultView {
+passed: bool | None
+message: str
+elapsed_seconds: float | None
+condition_statuses: list[ConditionNode]
+overrides: list[str]
+raw_log: str
+trajectory_path: str | None
}
class ConditionNode {
+label: str
+satisfied: bool
+message: str
+condition_type: str
+role: str
+details: dict
+children: list[ConditionNode]
}
class RunProgress {
+current: int
+total: int
+scenario_name: str
+status: Status
}
SessionSummary "1" --> "*" SessionItem : contains
SessionItem "1" --> "1" ScenarioResultView : details
ScenarioResultView "1" --> "*" ConditionNode : condition tree
ConditionNode "1" --> "*" ConditionNode : children (recursive)
Result Directory Structure¶
The scanner discovers results from two directory layouts:
base_path/
├── outputs/ # Single and batch runs
│ └── YYYY-MM-DD/
│ └── HH-MM-SS/ # HH-MM-SS-1, -2, ...: a later run in the same second
│ ├── ScenarioName_result.json
│ ├── ScenarioName_coverage.json # Coverage hits (docs/coverage.md)
│ ├── ScenarioName.trajectory.jsonl # Actor motion (docs/trajectory_log.md)
│ ├── ScenarioName.log # CARLA native recording
│ ├── batch_results.json # Batch summary (batch mode only)
│ └── .hydra/
│ └── overrides.yaml # Hydra CLI overrides
│
└── multirun/ # Sweep / multi-scenario runs
└── YYYY-MM-DD/
└── HH-MM-SS/
├── 0/ # Job index 0
│ ├── ScenarioName_result.json
│ ├── ScenarioName.trajectory.jsonl
│ ├── raw_output.log # Subprocess stdout/stderr (a viewer run's)
│ └── .hydra/
│ └── overrides.yaml
├── 1/ # Job index 1
│ └── ...
└── N/
Session types:
| Type | Source | Identification |
|---|---|---|
multirun |
multirun/ directory |
Numbered subdirectories (0, 1, 2, ...) |
batch |
outputs/ directory |
Contains batch_results.json |
single |
outputs/ directory |
Contains *_result.json (no batch file) |
Scenario Execution from Viewer¶
The viewer runs scenarios via subprocess to isolate the CARLA Python API (which can cause SIGSEGV) from the web server process:
sequenceDiagram
participant B as Browser
participant A as FastAPI
participant R as Runner (thread)
participant S as Subprocess (uv run scenario)
participant SQ as ScenarioQueue
participant SR as ScenarioRunner
participant C as CARLA Simulator
B->>A: POST /api/run {scenario, extra_overrides, sweeper, launcher, slurm}
opt The body is not JSON, not a JSON object, names no scenario, or a field has the wrong type
A-->>B: 400 {"status": "error", "message": ...}
end
alt With sweeper
A->>A: sweep_resolver.resolve_sweep()
Note over A: expand_config, as the CLI: constraints,<br/>route searches, ODD draws
opt A scenario fails to expand
A-->>B: 400 {"status": "error", "message": ...}
end
else Without sweeper
A->>A: Expand glob to scenario names
end
opt The pattern is refused or matches no scenario
A-->>B: 400 {"status": "error", "message": ...}
end
A->>R: start_run(overrides_list, extra_overrides)
Note over R: Marks the run as running and resets<br/>its progress under one lock
alt Another run in progress
A-->>B: 409 {"status": "busy"}
else Started
A-->>B: {"status": "started"}
end
B->>A: GET /api/run/progress (SSE)
loop For each job
R->>R: _set_progress(running)
R->>S: Popen(["uv", "run", "scenario", ...overrides]), output to a file
rect rgb(240, 248, 255)
Note over S,C: Isolated process — same execution<br/>pipeline as the scenario CLI command
S->>SQ: main() → build_scenario() → queue.add()
SQ->>SR: run_scenario(scenario)
SR->>C: Setup, tick loop, recording, cleanup
C-->>SR: ScenarioResult
SR->>SR: Write *_result.json + *.trajectory.jsonl
end
S-->>R: exit code (stdout/stderr written to raw_output.log as it ran)
R->>R: _set_progress(passed/failed)
A-->>B: SSE event {current, total, status}
end
R->>R: _set_progress(done)
A-->>B: SSE event {status: "done"}
Key design decisions:
- Process isolation: Each scenario runs in a separate
uv run scenariosubprocess. This prevents CARLA's C++ extension from crashing the web server. - Thread-safe progress: A global
RunProgressobject protected bythreading.Lockprovides progress state.start_run()checks for a run in progress and claims the run under that lock, so two runs never start at once and the stream never replays the previous run'sdone. - Job output on disk: a job's stdout and stderr go to a file as it runs, never held in memory:
raw_output.login its job directory when the run is grouped as a multirun (the scenario page shows its last 256 KB), a temporary file otherwise -- an ungrouped job's directory is Hydra's, not known to the runner -- removed once the job is over. A job directory left empty (a job stopped or not launched before Hydra wrote to it) is removed too, so it is not listed as a case. A failed job's warning logs the last 4 KB. - Checked run bodies:
/api/runand/api/run/previewread their body through one parser (app.parse_run_body): it must be JSON (a body that does not parse, cut off or garbled, is refused rather than read as no body), a JSON object,scenarioa string,extra_overridesa list of strings (null or absent: none),sweepera string or null,launcherlocal,slurmor null, andslurman object of text or number options./api/runalso needsscenario: an empty body, or one without it, would otherwise run*/*. Anything else is answered with 400 before a run starts or a batch is submitted, so a string of overrides is never run one character per argument (#1500). So is a request the body is read from but that cannot run: a scenario pattern that is refused or matches nothing, a sweep that fails to expand, or a SLURM option value that is not valid (/api/run/previewtoo), each with the same JSON body as before; a SLURM submission is refused with 503 where SLURM is not available and 409 while another is still being made (#1964). The SLURMresubmitaction refuses a body that is not JSON or not an object, or afailedthat is not a boolean, the same way (an empty one resubmits with no flags). - SSE streaming: The
/api/run/progressendpoint uses Server-Sent Events for real-time progress updates (polled at 0.5 s intervals). - Multirun grouping: When running multiple scenarios, jobs share a timestamped
multirun/directory with numbered subdirectories. The directory is the run's alone: one started in the same second as another run or a CLI sweep getsHH-MM-SS-1, thenHH-MM-SS-2, ...
Condition Tree Display¶
The viewer reconstructs the condition hierarchy from the JSON result for display:
pass[0](all_roads_visited) ← AndCondition (top-level)
├── StickyCondition(road_5) ← Wrapper
│ └── EntityLanePositionCondition ← Leaf
├── StickyCondition(road_12)
│ └── EntityLanePositionCondition
└── StickyCondition(road_8)
└── EntityLanePositionCondition
fail[0](ego_existence) ← EntityExistenceCondition
fail[1](default_timeout) ← TimeoutCondition
The scenario's own fail conditions come first, then the runner's
ego_existence, then default_timeout — which is absent when the scenario
registered a timeout of its own.
The _build_condition_tree() function in scanner.py recursively parses:
childrenkey for composite conditions (AndCondition,OrCondition)childkey for wrapper conditions (StickyCondition,PersistentCondition,NotCondition)
Caching Strategy¶
The scanner uses a module-level dictionary cache (_cache) keyed by path and session identifiers. The cache is cleared explicitly via the /api/refresh endpoint (triggered by the "Update" button in the UI). A batch's session and case pages are also kept only while the batch's files stay the same (cases.batch_stamp: the times of batch_results.json, the SLURM manifest, collected.json, and the slurm/submissions/ and slurm/results/ directories), so a SLURM batch resubmitted and collected again shows its new results on a reload; until it is collected again, its summary is older than its last submission or case outcome, and its cases are read from slurm/results/ instead. A multirun or single session is stamped too, since a run still in progress writes its files after its pages were first read (#1964): a multirun session's page by the modification times of its directory and of each job directory (one scandir and a stat per job, not per file: a job added, or a result, log or other file created, removed or renamed in a job changes one), and it is kept only once every job has a result that could be read (a result written in place changes no directory's time); a case page of a multirun job, and a single run's session and case pages, by the name, time and size of every entry of the run's directory (a handful of stat calls). This avoids repeated filesystem scans while keeping results fresh on demand.
Coordinate Transform System¶
Three coordinate systems are unified through MapManager:
graph LR
L2["Lanelet2Pose<br/>(lanelet_id, s)"]
OD["OpenDrivePose<br/>(road_id, lane_id, s)"]
CW["CarlaWorldPose<br/>(x, y, z, yaw)"]
L2 -->|"to_opendrive()"| OD
L2 -->|"to_carla_world()"| CW
OD -->|"to_carla_world()"| CW
CW -->|"to_lanelet2()"| L2
OD -->|"snap_to_carla_road()"| CW
CW -->|"to_opendrive()"| OD
MapManager is a singleton that holds the LaneletMap, RoadNetwork, and a precomputed z-offset (averaged from CARLA spawn points, leaving out those matched to a lanelet on another level of stacked roads). It is initialized once per ScenarioQueue.start() when both xodr_path and lanelet2_path are provided.
to_carla_world() returns a CarlaWorldPose; to_carla_location() takes any of the three (or a carla.Location) and returns the carla.Location of its CarlaWorldPose.
snap_to_carla_road() uses CARLA's ray-cast API to project an OpenDRIVE position onto the actual 3D road surface, accounting for terrain height differences between the map definition and the rendered world.
Entity Management¶
classDiagram
class VehicleEntityConfig {
+role_name: str
+spawn_location: SpawnLocation
+vehicle_type: str
+initial_speed_kmh: float
+spawn_retry_max_count: int
}
class EgoConfig {
+role_name = "Ego" (fixed)
+goal_pose: Lanelet2Pose | None
}
class SpawnLocation {
<<interface>>
}
class SpawnTransform {
+transform: carla.Transform
}
class SpawnPointIndex {
+index: int
}
class VehicleEntity {
+actor: carla.Actor | None
+spawn(world) carla.Actor
+destroy()
}
class EgoVehicle {
+use_autopilot = true
+requires_goal = false
+actor: carla.Actor | None
+spawn(world, config) carla.Actor
+on_scenario_start(world)
+on_tick(world, elapsed)
+on_scenario_end(world)
+destroy()
}
class ExternallyDrivenEgo {
+use_autopilot = false
+stack_name: str
+termination_requested: bool
+termination_reason: str | None
+change_lane() refuses
+set_desired_speed() refuses
+turn_at_junction() refuses
}
class CarlaDriverEntity {
+handshake: BridgeHandshake | None
}
class BridgeHandshake {
+bridge: AutowareBridge
+launcher: AutowareLauncher | None
+set_mission(initial_pose, goal_pose, waypoints)
+route_to(world, goal)
+start(actor)
+poll() reason | None
+ready: bool
}
class Ros2EgoEntity
class BridgedEgoEntity {
+requires_goal = true
+handshake: BridgeHandshake
+is_initialized: bool
}
class AutowareEgoEntity
VehicleEntityConfig <|-- EgoConfig
SpawnLocation <|.. SpawnTransform
SpawnLocation <|.. SpawnPointIndex
VehicleEntityConfig --> SpawnLocation
VehicleEntity --> VehicleEntityConfig
EgoVehicle <|-- ExternallyDrivenEgo
ExternallyDrivenEgo <|-- CarlaDriverEntity
ExternallyDrivenEgo <|-- Ros2EgoEntity
ExternallyDrivenEgo <|-- BridgedEgoEntity
BridgedEgoEntity <|-- AutowareEgoEntity
BridgedEgoEntity *-- BridgeHandshake
CarlaDriverEntity o-- BridgeHandshake
The ego hierarchy splits on who drives. EgoVehicle is driven by the traffic
backend. ExternallyDrivenEgo is the base of every ego a stack of its own drives:
it opts out of the TrafficManager, refuses TrafficManager manoeuvres instead of
delegating them, and carries the termination request. BridgeHandshake is the
splatsim-consistent AutowareBridge handshake — the mission handed
over, readiness awaited, the stack launched and stopped per scenario — which
depends only on the abstract bridge, so it is not particular to Autoware. An ego
holds one rather than inheriting it, as the static check takes single inheritance
only: BridgedEgoEntity is the ego that is nothing but the handshake (the stack
drives), AutowareEgoEntity is that with Autoware on the other end, and a
CarlaDriverEntity given a bridge holds one beside its policy loop, starting the
policy once the stack is ready.
Ego lifecycle hooks: ScenarioRunner calls on_scenario_start() after the warm-up
ticks, on_tick() on every simulation tick, and on_scenario_end() during teardown.
They are no-ops on EgoVehicle, so a TrafficManager-driven ego costs nothing. An ego
that drives itself overrides them — see
External Driver Interface.
Who drives the traffic: everything the scenario did not author belongs to a
traffic backend — traffic_manager by default, none for
an empty road, or one from another package. ScenarioRunner calls
prepare / start / tick / close on it and never names a traffic simulator
itself, and an entity's manoeuvre intents (change_lane, turn_at_junction,
follow_route) are delegated to whichever backend drives it. A goal given to a
vehicle the backend drives, the ego or an NPC, becomes one of them:
BackendDriven.route_to hands the backend the route from where the vehicle is,
through any waypoints, to it. A pedestrian given one lays its walk out on the
walkable lanelets (PedestrianEntity.route_to), and the RoutingAction walks
it a tick at a time (PedestrianEntity.walk_on, steered as a RoutingAction
steers a walker in follow mode) -- so on a pedestrian the action needs the
tick loop (BaseAction.needs_tick_loop). The walk is searched on
coordinate.walk_graph.WalkGraph, not on Lanelet2's routing graph: a crosswalk
is usually drawn across the walkways it joins, reaching onto them without
sharing a point, so Lanelet2 joins nothing across a road. The walk graph joins a
walkable lanelet's end to any walkable lanelet it lies on (or within 0.5 m of),
at that point along it, joins two walkable lanelets where their centrelines
cross (a crosswalk drawn right across a walkway), and walks every walkway both
ways whatever its one_way tag. A leg it has no walk for (a map with no
walkways, as most Autoware maps, or a waypoint off them) is a straight line,
with a warning.
Who drives the ego: use_autopilot decides whether the backend is given the
ego actor to drive or told to keep its hands off it. EgoVehicle opts in (TrafficManager drives);
every ExternallyDrivenEgo opts out: on its own it leaves the actor for something
outside the scenario, CarlaDriverEntity drives it from an external policy's plan,
Ros2EgoEntity from ROS 2 nodes' commands, and BridgedEgoEntity leaves it to the
stack it handed the mission to.
Where the ego is going: requires_goal says whether an entity can start without a
destination. A BridgedEgoEntity (AutowareEgoEntity) cannot — the goal is what makes it move at all — and
BaseScenario.require_goal(), asked by register_route_to_goal and by ScenarioRunner
once setup() returns, refuses one that has none. An ego driven by the TrafficManager or
a driver policy reads no goal, so a scenario may name none and the routing action is
simply not registered. The goal reaches the ego config from ego.goal_lanelet_id or from
the scenario itself — IntersectionPassingScenario derives it from the route it asserts,
through BaseScenario.derive_goal_from_route().
Spawn retry logic: When a vehicle fails to spawn (e.g., collision with existing geometry), the spawn system retries with lateral (t_step) and vertical (z_step) offsets, up to spawn_retry_max_count attempts.
Lanelet Constraint Sweeper¶
The sweeper enables automated parameter sweeps across lanelets that match specified constraints:
graph TB
subgraph Input
SC["Scenario Config<br/>(YAML)"]
Constraints["Sweep Constraints<br/>(traffic_light, direction, ...)"]
Bindings["Sweep Bindings<br/>(ego.spawn_s, npc.spawn_s, ...)"]
end
subgraph Processing
Map["Load Lanelet2 Map"]
RG["Create Routing Graph"]
Match["Find Matching Lanelets"]
Resolve["Resolve Bindings<br/>for each matched lanelet"]
end
subgraph Output
Jobs["Override Lists<br/>[scenario=X, ego.spawn_lanelet_id=5, ...]"]
end
SC --> Map
Constraints --> Match
Map --> RG
RG --> Match
Match --> Resolve
Bindings --> Resolve
Resolve --> Jobs
The sweeper is registered as a Hydra plugin (hydra_plugins/autoware_scenario_sweeper/) and can also be invoked directly from the viewer via sweep_resolver.py (without CARLA, using only the Lanelet2 map).
Ports and Environment Variables¶
Network Ports¶
The system uses three TCP ports for inter-process communication with the CARLA simulator and the viewer web server:
graph LR
subgraph Viewer["Viewer Process"]
FastAPI["FastAPI<br/>(Uvicorn)"]
end
subgraph Scenario["Scenario Process"]
Client["carla.Client"]
TM_Client["TrafficManager<br/>Client"]
end
subgraph CARLA["CARLA Simulator"]
RPC["RPC Server"]
TM_Server["TrafficManager<br/>Server"]
end
subgraph Browser["Browser"]
B["User"]
end
B -->|"TCP 9000<br/>(VIEWER_PORT)"| FastAPI
Client -->|"TCP 2000<br/>(server.port)"| RPC
TM_Client -->|"TCP 8100<br/>(traffic_manager.port)"| TM_Server
| Port | Default | Component | Protocol | Configured via | Description |
|---|---|---|---|---|---|
| 2000 | 2000 | CARLA RPC Server | TCP | server.port (Hydra override, e.g. scenario server.port=2001; scenario takes no --port) |
Main communication channel between carla.Client and the CARLA simulator. Used for world control, actor spawning, recording, etc. |
| 8100 | 8100 | CARLA TrafficManager | TCP | traffic_manager.port (Hydra), tm_port constructor arg |
RPC port for the TrafficManager, which controls NPC vehicle autopilot behavior. The CARLA default is 8000, but this project uses 8100 to avoid conflicts with other services (e.g., VS Code). |
| 9000 | 9000 | Viewer Web Server | HTTP | viewer --port, VIEWER_PORT env var |
FastAPI/Uvicorn server for the result viewer web UI. Serves HTML pages, REST API, and SSE progress stream. |
Note
The CARLA RPC port and TrafficManager port are used within the scenario subprocess, not the viewer process. The viewer communicates with CARLA only indirectly through subprocess.run(["uv", "run", "scenario", ...]).
Environment Variables¶
| Variable | Required | Default | Used by | Description |
|---|---|---|---|---|
CARLA_EXECUTABLE |
No | the launcher scenario-setup installed |
CarlaServerManager |
Absolute path to the CARLA UE5 launcher (CarlaUnreal.sh). Unset, the one scenario-setup installed under ~/.autoware_carla_scenario/bin/carla is launched. When a server is already running and reuse_if_running=True, neither is needed. Also a gate for pytest: tests are skipped when there is no launcher at all and no server answers at their host. |
AUTOWARE_CARLA_SCENARIO_HOME |
No | ~/.autoware_carla_scenario |
carla_install_paths |
Where scenario-setup installs CARLA (bin/carla). |
NISHISHINJUKU_MAP_PATH |
Yes (if map.overwrite_xodr is on) |
— | ScenarioRunner.load_map_by_overwriting_xodr() |
Path to the internal .xodr file inside the CARLA installation for the NishishinjukuMap. Used to overwrite the built-in OpenDRIVE file with a custom version while retaining full CARLA map assets (meshes, textures). The variable name is derived from the map name via CamelCase → UPPER_SNAKE_CASE_PATH conversion. |
NISHISHINJUKU_XODR_PATH |
No | data/nishishinjuku_carla.xodr |
Map config YAML (${oc.env:...}) |
Path to the custom OpenDRIVE file for the Nishishinjuku map. Resolved by OmegaConf's oc.env interpolation in conf/map/nishishinjuku.yaml. |
NISHISHINJUKU_LANELET2_PATH |
No | data/nishishinjuku.osm |
Map config YAML (${oc.env:...}) |
Path to the Lanelet2 .osm file for the Nishishinjuku map. Resolved by OmegaConf's oc.env interpolation. |
VIEWER_BASE_PATH |
No | Current working directory | Viewer (ui/__init__.py) |
Base directory that the viewer scans for outputs/ and multirun/ result directories. viewer --base-path wins over it. |
VIEWER_HOST |
No | 127.0.0.1 |
Viewer (ui/__init__.py) |
Bind address for the Uvicorn HTTP server. viewer --host wins over it; 0.0.0.0 serves every interface, with no authentication. |
VIEWER_PORT |
No | 9000 |
Viewer (ui/__init__.py) |
Listen port for the Uvicorn HTTP server. viewer --port wins over it. |
SWEEP_RESUME_FROM |
No | 0 |
Sweeper (lanelet_constraint_sweeper.py reads it; sweeper/job_status.py's resume_from_env() checks its value) |
Internal variable set by the --resume-from N CLI flag. Passed via environment because Hydra's CLI parser rejects unknown overrides. Tells the sweeper to start at job N (1-indexed); set (to a whole number, 0 or empty for none), it wins over sweep.resume_from, and any other value is refused (exit 2). A batch (a glob, or launcher=slurm) reads the flag directly instead; a single run or Hydra's basic sweeper, which nothing would resume, refuses the flag (exit 2). |
Dynamic Map Path Variables¶
ScenarioRunner.load_map_by_overwriting_xodr() runs only when the map config sets
overwrite_xodr: true (or ScenarioQueue(overwrite_xodr=True) is passed directly).
It is off by default: a map cooked for CARLA already ships the OpenDRIVE its assets
were built from, and replacing that puts the road network and the meshes out of step.
When it is on, the method derives the destination environment variable name from the
CARLA map name using map_asset_env_var() (maps/opendrive.py):
| Map Name | Derived Variable |
|---|---|
NishishinjukuMap |
NISHISHINJUKU_MAP_PATH |
Town01 |
TOWN01_PATH |
Town10HD_Opt |
TOWN10_HD_OPT_PATH |
The conversion rule is: insert _ before each uppercase letter that follows a lowercase letter or digit, convert to UPPER_SNAKE_CASE, and append _PATH.
Key Design Decisions¶
Process Isolation for Viewer¶
The viewer deliberately avoids importing CARLA in its own process. All scenario execution happens via subprocess.run(["uv", "run", "scenario", ...]). This prevents CARLA's C++ extension from causing segmentation faults in the web server.
Synchronous Simulation Mode¶
All scenario execution uses CARLA's synchronous mode at 20 Hz. This ensures:
- Deterministic tick ordering (actions, conditions, physics all advance together)
- Reproducible results with fixed random seeds
- No frame drops during recording
Nothing Rendered During a Run¶
A run renders no camera image for itself: the tick loop does no RGB processing beyond what the ego's own sensors need, which keeps condition evaluation timing accurate. What is needed to see the run afterwards is written as data (the trajectory log) and drawn by a viewer.
Condition Composition Pattern¶
Complex pass/fail criteria are built by composing atomic conditions:
# Example: pass when ego visits all three road segments
pass_condition = AndCondition(
label="all_roads_visited",
conditions=[
StickyCondition(EntityLanePositionCondition.anywhere_on_road("ego", "5", ...)),
StickyCondition(EntityLanePositionCondition.anywhere_on_road("ego", "12", ...)),
StickyCondition(EntityLanePositionCondition.anywhere_on_road("ego", "8", ...)),
],
)
StickyCondition latches once its child is satisfied, so each road only needs to be visited once. AndCondition requires all children to be satisfied simultaneously. It checks every child on every tick, so each sticky child latches when its own road is reached and the roads may be visited in any order; to require an order, latch the later one on the earlier: StickyCondition(AndCondition([StickyCondition(on_5), on_12])).
World Reset for Clean State¶
Between two scenarios -- before the next one's attempt (and before its cooldown), and before a retry -- the queue has ScenarioRunner.reset_world() reset the world; the last scenario's world is reset by the next run_all(), or, on a server the queue does not stop, by stop() (ScenarioRunner.restore_world(), #2067). By default (server.world_reset: reload) it loads the batch's map again by name (client.reload_world() fails on CARLA UE5): a new episode, with the run's actors destroyed first so their sensor streams are released. server.world_reset: soft keeps the episode and puts back what a run changes -- destroying every actor the freshly loaded world did not have and restoring the settings, weather, traffic lights and spectator read right after the load (world_reset.py) -- and falls back to a reload when the world is another episode or anything is left over. See Resetting the world between cases for what a soft reset does and does not reset.