Skip to content

Running a Batch on SLURM

A batch (scenario 'scenario=<glob>', and with --multirun every case of each scenario's sweep) runs its cases one after another, in one process, on one CARLA. On a machine or cluster with SLURM, the same batch can run its cases as the tasks of a SLURM array job instead, several at once, each on a CARLA of its own:

uv run scenario --multirun 'scenario=cut_in/*' +sweep.odd_sample.count=24 \
  launcher=slurm launcher.array_parallelism=2 launcher.gres=shard:1 launcher.mem=12G

launcher=slurm needs scenario=: without it there is no batch to submit, and scenario launcher=slurm is refused (Error: launcher=slurm runs a batch: ..., exit 2) rather than running the default scenario locally.

It is off unless a run selects it. The launcher config group is not part of the default config: a run that does not name it composes the same config as before, and the batch runs here, exactly as it always has. launcher=local says the same explicitly.

The batch writes what a local batch writes, where a local batch writes it (outputs/<date>/<time>/), under the same names: every case's {Scenario}_result.json, {Scenario}.trajectory.jsonl, coverage, ODD archive, environment and CARLA recording, and batch_results.json with the summary table at the end. A case's outputs do not depend on which task ran it or when.

How it works

On the machine you run scenario on (a SLURM submit host):

  1. The batch is expanded, labelled, built and statically checked exactly as a local batch is: the ODD sampling, the case labels, the static check (once a config shape), the map fetched. A config error is refused here, before anything is submitted, and so is a scenario whose sweep expands to no case (Error: <scenario>: the sweep expands to no case ..., exit 1).
  2. A manifest is written into the batch's output directory (slurm/manifest.json): every case's scenario and overrides, its resolved config, and the file stem it is to be named by.
  3. sbatch --array=0-(N-1)%K submits the cases, launcher.cases_per_task to a task, at most launcher.array_parallelism tasks at once. An sbatch that has not returned after 120 s (a controller that does not answer) is stopped, and the submit fails with sbatch timed out after 120.0 s. Unlike a refused sbatch, whose submission record is removed, the controller may have taken this job all the same, so its record is kept with no job id, and says that sbatch timed out, and when (sbatch_timed_out, sbatch_timed_out_at): if squeue --me lists the job, its tasks find their cases and run, and scenario-slurm collect collects the batch once it ends (cancel knows no job id to cancel: scancel it by hand). Its index is not given to another submission, and the record counts as in flight, so a resubmit of the batch is refused until 600 s (SBATCH_IN_FLIGHT_SECONDS) after it was made, saying that its sbatch timed out and to check squeue --me; after that a resubmit is taken -- one that would run the cases a still-queued job also runs, so check squeue --me first. status and the viewer show such a submission as no job id: sbatch timed out at <time>; the job may exist all the same: check squeue --me, its cases unknown in the viewer (not missing), and wait says it has no job id to follow and where to look for the job (and, within those 600 s, that a resubmit is refused); collect reports each of its cases as task <n> of submission <i> never started with the same note, never job None, and (within those 600 s) that a resubmit is refused.
  4. By default scenario follows the array (squeue, and the outcome each task records) until it ends, then prints the summary and writes batch_results.json, as a local batch does, and exits 0 when every case passed. Ctrl-C cancels the batch's jobs, waits (up to 60 s) for their tasks to leave the queue, and collects. With --no-wait (launcher.wait=false) it submits and exits, and a small job that runs once the array has ended (--dependency=afterany) collects the batch; should that job not be submitted, scenario says so, with the array's job id, and exits non-zero (scenario-slurm wait collects the batch instead).

Each task, on its node:

  1. takes a slot of ports no other task on the node holds and nothing listens on (Ports);
  2. composes each of its cases' configs from the manifest's scenario and overrides, and refuses to run one that no longer resolves to the config it was submitted with (a job that does not run in the submitting environment);
  3. builds them (not checked again: the submitting process did), starts a CARLA on its ports and runs its cases one after another through the same queue a local batch uses, writing their files into the batch's directory under the names a local batch gives them -- each Autoware stack's log named after its case's stem (autoware-<stem>.log), as a local batch names it, and a containerised backend's log after the task (backend_<name>-<array job id>_<task>.log), so tasks never append to one file (#2247). What a local batch decides once for all its cases, every task decides the same, however the cases are split into tasks (cases_per_task): the queue's settings (server, traffic, t4, t4scene, world reset, backends) are the batch's first case's, Rerun recordings are made when any case sets rerun.enabled (strict when any sets rerun.strict), and the canvas and the heavy metrics are the batch's. The manifest records which cases decide them (batch_settings), and a task composes those beside its own;
  4. records each case's outcome as soon as it has run (slurm/results/<case>.json), and its own progress in slurm/tasks/<submission>-<task>.json (host, ports, state, peak resident memory); with rerun.enabled=true it also converts each case's trajectory log into its Rerun recording ({stem}.rrd) and records whether that failed, so collect reports the recordings as a local batch does (Seeing a run). The submitting process only checks that the rerun extra is installed; the nodes need it too;
  5. stops its CARLA however it ends: at the end, on scancel, on the signal SLURM sends launcher.signal_grace_seconds before launcher.time runs out, and -- through the job script -- when the worker itself was killed, even by the SIGKILL that ends a scancel after SLURM's KillWait (CARLA runs in a session of its own, which that signal misses): a small watchdog process of its own, started once the slot is taken, stops CARLA when the worker ends without having done so, and keeps the slot locked until then.

Following, collecting, resubmitting

scenario-slurm takes the batch's output directory:

scenario-slurm status   outputs/2026-10-18/12-00-00   # cases, jobs, tasks
scenario-slurm wait     outputs/2026-10-18/12-00-00   # follow, then collect
scenario-slurm collect  outputs/2026-10-18/12-00-00   # summary, batch_results.json
scenario-slurm resubmit outputs/2026-10-18/12-00-00   # the cases with no result
scenario-slurm cancel   outputs/2026-10-18/12-00-00   # scancel the batch's jobs

cancel gives scancel 60 s; one that has not returned by then is stopped, cancel says so on one line and exits 1. The same 120 s limit on sbatch, and the kept record, as above hold for resubmit; should the collect job's sbatch time out, the array runs all the same and scenario says how to collect it by hand.

status, collect and cancel take nothing but the batch directory. The options of the others (scenario-slurm <command> --help says the same):

Command Option
wait --submission K the submission to follow, by its index (slurm/submissions/<K>.json; 0 is the first); default: the latest
wait --poll-seconds S seconds between two polls of squeue; default: the batch's launcher.poll_seconds. Below 0.1 is taken as 0.1, a negative number is refused
resubmit --failed also the cases that ran and failed
resubmit --cases a,b these cases instead, by label or index; a case named twice runs once
resubmit --array-parallelism K at most this many tasks at once (at least 1); default: the batch's launcher.array_parallelism
resubmit --cases-per-task N cases a task runs (at least 1); default: the batch's launcher.cases_per_task
resubmit --wait, --no-wait follow the new submission until it ends and collect, or submit and exit; default: the batch's launcher.wait. With nothing to resubmit, it collects at once, but only with an explicit --wait (and not with --dry-run)
resubmit --force submit even while a job of the batch is queued or running, squeue cannot tell whether one is, or another submission of the batch is in flight or its sbatch timed out less than 600 s after its record was made
resubmit --dry-run print the cases it would resubmit, submitting and collecting nothing, even with --wait, and exit 0; it still exits 1 where a resubmit would refuse: a busy batch without --force, an unknown --cases entry or an out-of-range option, and on an error reading the batch

A task that raised, every failed attempt of a case and a native crash in a task leave a crash report in the batch's crash_reports/, as a local batch's do. A case's report runs the case again on its own, with scenario and without launcher=slurm; a failed task's status names its report (crash_report).

A case that has no outcome -- its task was cancelled, ran out of time, was killed for memory, or its node failed -- is reported as ERROR with a message saying so, and collect lists them, saying what to do about them, chosen from the batch's records alone (#1803): where the case's task may still be queued or running (it has a job, and it has no status yet, or its status says starting, waiting, building or running), to wait for it with scenario-slurm wait (with --submission <k> when its submission is not the latest) if squeue --me still lists it and to resubmit once it has ended; where its submission is in flight, to try again once sbatch returns; where its sbatch timed out less than 600 s ago, to collect once the job ends if squeue --me lists it, and otherwise to resubmit, which is refused until those 600 s have passed unless with --force; and otherwise to resubmit. So is a case whose recorded outcome holds no verdict -- no result object, or a passed that is not true or false ("false" and 1 are none) -- or a result that cannot be reported: a message that is missing or not a string, an elapsed_seconds that is not a number (true is none), condition_statuses that are not a list of objects each with a string label and message and a boolean satisfied, or a failure that is not an object or cannot be read (an at of the wrong type, say). What the worker writes always is readable; rerun_recording and failure may be missing. It did not pass, and collect reports it as ERROR naming its file; a rerun_error the task recorded for it is kept in batch_results.json. status, wait and the viewer count it as not passed: the results pages show it as an error. An outcome is stale when the last submission that gave the case was to run it again and its task stopped (failed or cancelled) before it reached the case: the task removes a case's outcome only right before it runs the case, so the one there names an earlier submission. collect reports it as ERROR saying so, and status and the viewer count it as without a result (#2247). resubmit submits again the cases with no outcome, no verdict or a stale one and those that raised on every attempt (ERROR); --failed adds the ones that ran and failed, --cases a,b names them (labels or indices; a case named twice runs once), and --array-parallelism/--cases-per-task change how they are spread. A case run again replaces its earlier files: before it runs, the task removes every output of its stem (run_outputs), its T4 dataset directory ({stem}.t4) included, so none of the last attempt's is left beside the new result. So resubmit refuses while a job of the batch is queued or running, while squeue cannot tell whether one is, or while another submission of the batch is in flight (--force submits all the same).

A batch directory that scenario-slurm cannot read or write -- a manifest it may not read, or shared storage mounted read-only -- is one Error: ... line naming the file, and the command exits 1, as for a directory that holds no batch. collect (and wait, and resubmit --wait, which end in it) prints the summary's verdict even when batch_results.json cannot be written, and writes slurm/collected.json after its hints, so the verdict is not lost (#1740).

Two resubmits of one batch may run at once -- from two viewer tabs, or from the viewer and scenario-slurm resubmit. Each submission's record, slurm/submissions/<k>.json, is created exclusively before sbatch runs, which reserves its index k: a submitter that finds k taken takes the next one, so no record is written over and no job id lost. The record is hard-linked into place from a temporary file (O_CREAT | O_EXCL where the file system has no hard links), which is atomic on NFS too, where flock may not be; a reader never sees it empty. Until sbatch returns, its job_id is null, and the submission counts as in flight: having reserved its index, a resubmit checks the submissions before it once more, and refuses should one be in flight or queued, so of two resubmits at once the later one submits nothing. A record whose sbatch fails is removed. One left with no job id by a submitter that was killed stops counting as in flight 600 s (SBATCH_IN_FLIGHT_SECONDS) after it was made. wait follows a submission only once it has a job id: on one still in flight it asks squeue nothing, collects nothing and exits non-zero, saying to try again once sbatch has returned; on one whose sbatch timed out, that the job may exist (check squeue --me, then collect or resubmit); and on one left with no job id it says the record is stale. status asks squeue nothing about a submission with no job id either: it never reads as a job that ended, but as no job id yet (sbatch in flight), as one whose sbatch timed out (above), or as a stale record; the viewer shows its cases pending, unknown or missing respectively. A record that cannot be read (empty, or not a JSON object) refuses a resubmit for as long after the file was last modified, then is ignored with a warning. The other readers skip such a record with a warning naming it: status lists it as unreadable, cancel says it skipped it (cancel its job with scancel if it had one), collect counts its cases as not submitted by it, and the viewer's batch page shows the batch with a note on that submission. collect can be run any number of times.

sacct is not needed: the batch is followed through squeue and the files the tasks write.

The result viewer does the same from its pages: its run panel submits a batch with launcher=slurm, and a SLURM batch's page shows its cases, jobs and tasks as they run and resubmits, collects or cancels it (Running through SLURM).

Configuration

launcher=slurm selects conf/launcher/slurm.yaml; every key can be set on the command line (launcher.mem=12G), or in a config.

Key Default
partition, account, qos null sbatch --partition/--account/--qos; partition may list several (gpu,cpu)
time null sbatch --time in any of its forms (90, 120:00, 168:00:00, 2-00, 1-12:30, 1-00:00:00, UNLIMITED); the task is sent SIGTERM signal_grace_seconds before it runs out
mem, cpus_per_task, gres null sbatch --mem/--cpus-per-task/--gres (gpu:1, or shard:1 to share a GPU; a list: gpu:1,shard:2)
array_parallelism null at most this many tasks at once (--array=...%K); null: no cap
cases_per_task null cases a task runs on its CARLA; null: the cases spread evenly over array_parallelism tasks (one case a task without a cap), in no more tasks than the cluster's MaxArraySize (Array size)
job_name null acs-<date>-<time> of the output directory
extra_sbatch_args [] further sbatch arguments; --clusters/--cluster/-M is refused (see below)
export null sbatch --export; null: ALL, or NONE with remote_cwd
env_passthrough null variables recorded at submission and set again in the job (a matched name no shell takes, such as an exported bash function, is left out); null: see below, or none with remote_cwd
setup [] shell lines the job runs before the worker (module load, source .venv/bin/activate); each is one command, and one ending in an unescaped backslash is refused
env {} variables set in the job, last; a name must be a shell variable name ([A-Za-z_][A-Za-z0-9_]*), as the run panel requires
python null the interpreter the jobs run (the tasks and the collect job): an absolute path, a path relative to the job's working directory (.venv/bin/python), ~/... (the home on the node), or a command on the node's PATH (python3); null: the submitting one
remote_cwd null where the jobs work on the nodes, for a submit host that sees their file system at another path (A submit host with other paths); null: the directory scenario runs in
headless true unset DISPLAY in the job, so CARLA renders off-screen
start_stagger_seconds 0 no task of the batch starts sooner than this after another did: the first wave, the tasks started as slots free up, a resubmit's (Shared state)
wait true follow and collect; false (or --no-wait): submit and exit
poll_seconds 15 how often wait asks squeue
cancel_on_interrupt true Ctrl-C while waiting cancels the batch's jobs
collect_job true with wait: false, a job that collects once the array ends
collect_mem 2G its --mem
signal_grace_seconds 120 --signal=B:TERM@N; at least 0
port_base, port_slots 20000, 1000 where the tasks' port slots are (Ports); port_base at least 1024, port_slots at least 1, and every slot's ports within 65535
port_lock_dir null where the slots' locks are; null: /dev/shm/acs-slurm-ports-<uid> (per user, so slots are exclusive among your own tasks only), else /tmp/acs-slurm-ports-<uid>, each used only if private; users sharing nodes set one group directory (Ports)

Each value is checked before anything is written, and one of the wrong type is refused with an error naming its key (launcher.<key>), never quietly read as something else:

  • The counts (cpus_per_task, array_parallelism, cases_per_task), signal_grace_seconds, port_base and port_slots are whole numbers: 2, 2.0 or "2", not 2.7, true, NaN or infinity. The counts and port_slots are at least 1.
  • The seconds (start_stagger_seconds, poll_seconds) are finite numbers, at least 0, not true.
  • The flags (wait, headless, collect_job, cancel_on_interrupt) are true or false, or one of their spellings as text ("false", "no", "off", "0", and "true", "yes", "on", "1"); "false" is false.
  • The text keys are text or a number, not a boolean, a list or a mapping; extra_sbatch_args, env_passthrough and setup are lists (or one string) of non-blank text, without a null or blank item; env is a mapping (or null, no variables) whose values are text or a number, not null or a boolean (quote one: FLAG: "true").
  • extra_sbatch_args may not send the batch to another cluster (--clusters, --cluster, -M, in any spelling: --clusters=x, -Mx): squeue, scancel and the collect job's dependency ask the default cluster, so following, resubmit's check that the batch is not running, and cancel would miss it. Should sbatch report a cluster all the same (SBATCH_CLUSTERS in the environment), the submission fails, naming the job to scancel -M (#2247).
  • 0, false or [] is a value, checked as any other, never read as unset; launcher.type is local only when it is missing or null ("", false or 0 is refused).

cases_per_task trades a task's start -- CARLA takes about 50 s to start, and a driver policy about as long to load -- against how evenly the cases spread and how many a lost task takes with it.

Array size

SLURM caps an array's size: its task ids run from 0 to one less than the cluster's MaxArraySize (scontrol show config; 1001 by default), so an array holds at most MaxArraySize tasks. Without cases_per_task the launcher reads MaxArraySize from scontrol show config (1001 when scontrol cannot be asked or names none) and puts as many cases in a task as it takes to stay within it: 2500 cases on the default limit run in 834 tasks of up to 3 cases. A cases_per_task you set is taken as it is; should it make more tasks than MaxArraySize, nothing is submitted and the error says the cases_per_task that fits. Should sbatch still refuse the array (a limit scontrol did not report), its error says to set launcher.cases_per_task.

The job's environment

A task runs with the Python the batch was submitted from (sys.executable, or launcher.python) and its environment:

  1. sbatch --export=ALL (launcher.export) hands the job the environment of the shell sbatch runs from;
  2. the job script then sets again, as they were when the batch was submitted, the variables launcher.env_passthrough matches: PATH, LD_LIBRARY_PATH, PYTHONPATH, PYTHONHOME, VIRTUAL_ENV, CONDA_PREFIX, CARLA_* (CARLA_EXECUTABLE), AUTOWARE_*, ONEPLANNER_*, TYPESAFE_CARLA_*, GLIBC_TUNABLES, MALLOC_*, CUDA_HOME, CUDA_PATH, CUDAToolkit_ROOT -- so scenario-slurm resubmit from another shell runs the cases as they were submitted;
  3. launcher.setup runs;
  4. launcher.env is set.

The collect job (launcher.collect_job) runs its script, slurm/jobs/<submission>-collect.sh, with the same environment: steps 1 to 4, then launcher.python in the directory the batch was submitted from, so a setup that loads a module or activates a venv serves it as it does the tasks.

A driver policy's variables belong on env_passthrough (a pattern: launcher.env_passthrough=[...] replaces the list) or in launcher.env. No token or credential is on the default list, and a name with TOKEN, SECRET, KEY, PASSWORD or CREDENTIAL in it (in any case) is not recorded through a pattern (AUTOWARE_*) either -- only when env_passthrough names it exactly, or launcher.env sets it: the manifest and the job scripts holding the values are files in the batch's output directory (both the user's alone, mode 0600 and 0700). With export: NONE, the job has only what steps 2 to 4 give it.

With launcher.remote_cwd set, none of this comes from the submit host: the job has the environment SLURM gives it on the node (export: NONE), the variables env_passthrough names only when it is given, then setup and env (A submit host with other paths).

A config that reads the environment (${oc.env:ONEPLANNER_MAP_DIR}) is resolved in the task too; a case whose config resolves differently there than where it was submitted is refused, naming the keys that differ.

A submit host with other paths

By default the jobs run where scenario was run: the directory it ran in, the interpreter it ran with and the batch's output directory are recorded by their absolute paths, which the nodes must see as the submit host does. A submit host that sees the nodes' file system somewhere else -- a workstation that mounts a node's ~/workspace/OnePlanner with sshfs or NFS at ~/mnt/node/OnePlanner, under another user's home -- names the nodes' paths instead, none of them absolute:

# On the submit host, in its mount of the node's ~/workspace/OnePlanner:
cd ~/mnt/node/OnePlanner
uv run scenario --multirun 'scenario=cut_in/*' +sweep.odd_sample.count=24 \
  launcher=slurm launcher.partition=main launcher.gres=shard:1 \
  launcher.remote_cwd=workspace/OnePlanner \
  launcher.python=.venv/bin/python \
  "launcher.setup=['source ~/workspace/opt/env.sh']"
  • launcher.remote_cwd is where every job -- each task and the collect job -- works on the node. A relative path is relative to the job user's home on the node, ~/... is that home too, and an absolute path is used as it is. It is expanded by the job script on the node, never here: the home is the one the node's passwd database gives the job's user (whatever $HOME the job was handed), so the user name and home may differ from the submit host's.
  • launcher.python is then required: the submit host's interpreter is not the node's. A relative path (.venv/bin/python) is relative to remote_cwd, a bare name (python3) is looked up on the node's PATH as setup leaves it.
  • The batch's output directory must lie under the directory scenario runs in (outputs/<date>/<time>/ does), which is what the node sees at remote_cwd; scenario refuses a batch elsewhere. The manifest records it relative to that directory (batch_dir_relative, a version 2 manifest), and the jobs find it under their own working directory. On the submit host, scenario-slurm takes the batch's directory as this host sees it (scenario-slurm wait ~/mnt/node/OnePlanner/outputs/<date>/<time>), and resubmit submits with the settings the batch was submitted with.
  • Nothing of the submit host's environment reaches the job: export is NONE (the job starts with the environment SLURM gives it on the node, which on most clusters is the user's login environment) and env_passthrough is empty, unless either is set. Set the node's environment with setup, which runs in remote_cwd and may read unset variables (source ~/workspace/opt/env.sh, module load cuda, source .venv/bin/activate), and env.
  • SLURM writes a job's output before the job knows where the batch is: each job starts in /tmp with its output there (/tmp/acs-slurm-<user>-<job>_<task>.out on the node) and sends it on to the batch's slurm/logs/ once it is in remote_cwd. The file in /tmp keeps what came before (a remote_cwd the node does not have) and what SLURM itself writes (a time limit, a cancel).
  • A case's config that resolves to absolute paths -- an ODD file in a scenario package, a map -- resolves to the node's paths there; see below for when that is still the same config.

Without remote_cwd the batch is submitted and run exactly as before (a version 1 manifest).

A value that is a path may differ and still be the same config. That is the case when it is the submitting directory's path seen from the job's working directory (<cwd>/x there, <remote_cwd>/x here), or when it names a file with the content it had where the batch was submitted: scenario records the SHA-256 of every file an absolute path in a case's config names (the manifest's files), and the task compares the file its own config names. A file missing on the node, or with other content, is refused, naming the key and both paths. This holds with or without remote_cwd: a package installed in another environment names its files at other paths too.

Ports

Each task's CARLA needs ports no other CARLA on the node uses, and a user may run one on the defaults (2000, TrafficManager 8100). A task takes a slot of 10 ports from port_base:

Port
base+0 CARLA RPC (server.port)
base+1 CARLA streaming (-carla-streaming-port)
base+2 CARLA secondary (-carla-secondary-port)
base+3 TrafficManager (traffic_manager.port, and the traffic backend's own: traffic.options.port, sumo's traffic.options.tm_port)
base+4 the Autoware bridge the run hosts (autoware.address)
base+5 the driver bridge the run hosts (driver.bridge.address)

The slot a task tries first is its array job id plus its task id, so the tasks of an array start on different slots. It takes one only when no other task using the same lock directory holds it (an exclusive lock, <lock dir>/autoware_carla_scenario-port-<base>.lock, which goes when the task's process does) and none of its ports is bound; else the next. The lock directory must be one every job on the node sees, which /tmp is not where SLURM gives each job a private one (job_container/tmpfs): it is /dev/shm/acs-slurm-ports-<uid> (or /tmp/acs-slurm-ports-<uid> where that cannot be made), or launcher.port_lock_dir. The default range, 20000-29999, is below Linux's ephemeral ports.

The default lock directory is per user, so slots and ROS domains are exclusive among your own tasks only. Another user's task on the same node may take the same slot: the check that none of its ports is bound stops that only once the other task's CARLA has bound them, tens of seconds after it took the slot. Users who share nodes set launcher.port_lock_dir to one directory they all write to, for example a group's own:

mkdir -p /dev/shm/acs-slurm-ports-mygroup
chgrp mygroup /dev/shm/acs-slurm-ports-mygroup
chmod 2770 /dev/shm/acs-slurm-ports-mygroup   # group-only, setgid, no sticky bit

The lock files are created 0644 less your umask, so keep a umask that leaves them readable by the group (022, 002 or 027). Do not use a world-writable directory: anyone on the node could then create or hold the lock files and keep your tasks off every slot. That is also why the default is not shared.

For the same reason a default directory is used only if it is private: a real directory (not a symlink) that you own and that neither your group nor others can write to. One another user made first, or one open to writing, is passed over with a warning for /tmp/acs-slurm-ports-<uid>, checked the same way; if that is not private either, the task stops with an error naming both. Have the node's administrator remove a directory another user made (chmod go-w one of your own), or set launcher.port_lock_dir. A configured port_lock_dir is taken as it is, as it may be shared on purpose. Lock files are never opened through a symlink.

The TrafficManager's port is always the slot's, whatever the config says: the traffic groups interpolate the backend's own (traffic.options.port, sumo's traffic.options.tm_port) from traffic_manager.port, and one set outright (traffic.options.port=8101, or null for the backend's default) is replaced by the slot's too, with a warning naming the key; otherwise every task on the node would ask its CARLA for a TrafficManager on that one port.

driver.address is not changed: a policy the run serves itself (driver.policy) listens on a port the OS picks, and one it does not is a server someone else runs, which every task dials. driver.transport is not changed either: with direct a task calls the policy it serves without a socket, and with unix it serves it on a socket in a directory of its own (under $TMPDIR, which SLURM does not set by default, so usually /tmp), so neither needs a port of the slot and tasks sharing a node never meet; a unix: driver.address someone else serves is dialled by every task, as a TCP one is.

ROS domains

The ROS 2 nodes of tasks on one node would share a DDS domain: they all start from the same config. So each task maps every domain id its config names to one no other task on the node holds (slurm/ports.py, RosDomains), and keeps it for all its cases:

Config What the task changes
ego.entity=ros2 ros2.domain_id
autoware.launcher.type=docker each autoware.launcher.ros_domain_ids entry; with none, one taken from autoware.launcher.env.ROS_DOMAIN_ID, else 0
autoware.launcher.type=command ROS_DOMAIN_ID in autoware.launcher.env (from its own value, else the job's exported ROS_DOMAIN_ID, else 0)

An id maps to the first domain from it on (up to 232, then from 0) whose lock, <lock dir>/autoware_carla_scenario-ros-domain-<id>.lock beside the port slots', no other task holds; the lock goes when the task's process does. The first task on a node gets the ids it asked for, the next ones the ids after: with ros_domain_ids=[10,11], one task runs on 10 and 11 and the next on 12 and 13. The ids are checked as a local run checks them (whole numbers from 0 to 232; a string ros_domain_ids is refused), naming the key.

A task whose domain is not the one configured logs a warning naming both and records the mapping in its progress file (slurm/tasks/<submission>-<task>.json, ros_domains, configured id to domain, beside ports). With ego.entity=ros2 that matters: ROS 2 nodes started outside the framework must join the task's domain (ROS_DOMAIN_ID=<domain>), not the configured one, so read it there, or give each such task a node to itself.

The domains are not part of the port slot: a slot's index runs to port_slots (1000 by default), and ROS 2 has 233 domains. Keep array_parallelism times the domains a task uses well below that, and below 102 on Linux (0 to 101 is what the ROS 2 documentation calls safe). The locks are per user (the lock directory is), so another user's tasks on the node may still pick the same domain, and nothing checks whether a domain is in use: users who share nodes share a lock directory (Ports).

Autoware stacks

The docker launcher labels each container it starts with your uid and with its owner, autoware_carla_scenario.owner=<host>:<pid>. A task removes only the leftover containers whose owner no longer runs, so a live stack of another task on the node is left alone (autoware.md). Tasks that build one workspace (build: always, or auto while it is unbuilt) do so one at a time, under a lock in the workspace: a task that waits for another's auto build uses it rather than build again.

Limits

  • Memory. A task runs a CARLA (about 6 GB resident) and a scenario process (about 6 GB with a TensorRT driver policy loaded). Without cgroups, SLURM uses --mem only to schedule, so set it, and array_parallelism, to what the node has.
  • Shared state. The tasks share the map caches (~/autoware_data/maps), the map files a driver policy reads (driver.map_dir) and the batch's output directory, so these must be on a file system every node sees. Map files, and the OpenDRIVE a run captures from the loaded world on a map's first run, are written atomically, so tasks that fill a cache at once do not break it. A driver policy's own caches are its own: one that two processes cannot load at once (OnePlanner's TensorRT engines, whose identity file is written under one temporary name, so two loads at the same moment fail one case with filesystem error: cannot rename) needs start_stagger_seconds, enough for one task to load it before the next starts (30 s for OnePlanner, which loads it once a task). The stagger is held with flock on slurm/start.lock, or where the file system has none, with a lock file beside it created with O_EXCL. One a killed task left is taken over once it is start_stagger_seconds + 60 s old by the file server's clock, so a node whose clock is off does not break a live one. One waiter takes a stale lock over: the waiters share a mutex of .takeover.* files beside the lock, one per attempt, created with O_EXCL (a waiter killed holding one is superseded by the next attempt 10 s later), and the one that wins judges the lock stale again before removing it. A task removes the lock on release only while it is still its own. So no waiter removes a live lock, and no two tasks start together, except when a task is still alive past the point its file is judged stale: one that held the lock that long, or a waiter that stalled for 10 s in the middle of taking a lock over.
  • ROS 2 and Autoware egos (ego.entity=ros2, ego.entity=autoware with a launcher) run their stacks on DDS. Each task takes ROS domains of its own (ROS domains), and its Autoware containers are told apart from another task's (Autoware stacks), so tasks sharing a node do not meet. A command launcher's script that sets ROS_DOMAIN_ID itself rather than reading it from its environment is the exception.
  • File names. A case is named by its position in the batch, which is the name a local batch gives it whenever every earlier case opened its trajectory log. A case that raises before it opens it (while its ego is created or the world is set to synchronous mode) leaves its name to the next case in a local batch; under SLURM every case keeps its own.
  • build_scenario_fn. A batch run from code with a build_scenario_fn is run by the tasks only when that function can be imported by name.