Running a Batch on SLURM¶
A batch (scenario 'scenario=<glob>', and with --multirun every case of each
scenario's sweep) runs its cases one after another, in one process, on one
CARLA. On a machine or cluster with SLURM, the
same batch can run its cases as the tasks of a SLURM array job instead, several
at once, each on a CARLA of its own:
uv run scenario --multirun 'scenario=cut_in/*' +sweep.odd_sample.count=24 \
launcher=slurm launcher.array_parallelism=2 launcher.gres=shard:1 launcher.mem=12G
launcher=slurm needs scenario=: without it there is no batch to submit,
and scenario launcher=slurm is refused (Error: launcher=slurm runs a
batch: ..., exit 2) rather than running the default scenario locally.
It is off unless a run selects it. The launcher config group is not part
of the default config: a run that does not name it composes the same config as
before, and the batch runs here, exactly as it always has. launcher=local
says the same explicitly.
The batch writes what a local batch writes, where a local batch writes it
(outputs/<date>/<time>/), under the same names: every case's
{Scenario}_result.json, {Scenario}.trajectory.jsonl, coverage, ODD archive,
environment and CARLA recording, and batch_results.json with the summary
table at the end. A case's outputs do not depend on which task ran it or when.
How it works¶
On the machine you run scenario on (a SLURM submit host):
- The batch is expanded, labelled, built and statically checked exactly as a
local batch is: the ODD sampling, the case labels, the static
check (once a config shape), the map fetched. A config error
is refused here, before anything is submitted, and so is a scenario whose
sweep expands to no case (
Error: <scenario>: the sweep expands to no case ..., exit 1). - A manifest is written into the batch's output directory
(
slurm/manifest.json): every case's scenario and overrides, its resolved config, and the file stem it is to be named by. sbatch --array=0-(N-1)%Ksubmits the cases,launcher.cases_per_taskto a task, at mostlauncher.array_parallelismtasks at once. Ansbatchthat has not returned after 120 s (a controller that does not answer) is stopped, and the submit fails withsbatch timed out after 120.0 s. Unlike a refused sbatch, whose submission record is removed, the controller may have taken this job all the same, so its record is kept with no job id, and says that sbatch timed out, and when (sbatch_timed_out,sbatch_timed_out_at): ifsqueue --melists the job, its tasks find their cases and run, andscenario-slurm collectcollects the batch once it ends (cancelknows no job id to cancel:scancelit by hand). Its index is not given to another submission, and the record counts as in flight, so aresubmitof the batch is refused until 600 s (SBATCH_IN_FLIGHT_SECONDS) after it was made, saying that its sbatch timed out and to checksqueue --me; after that aresubmitis taken -- one that would run the cases a still-queued job also runs, so checksqueue --mefirst.statusand the viewer show such a submission asno job id: sbatch timed out at <time>; the job may exist all the same: check squeue --me, its casesunknownin the viewer (notmissing), andwaitsays it has no job id to follow and where to look for the job (and, within those 600 s, that aresubmitis refused);collectreports each of its cases astask <n> of submission <i> never startedwith the same note, neverjob None, and (within those 600 s) that aresubmitis refused.- By default
scenariofollows the array (squeue, and the outcome each task records) until it ends, then prints the summary and writesbatch_results.json, as a local batch does, and exits 0 when every case passed. Ctrl-C cancels the batch's jobs, waits (up to 60 s) for their tasks to leave the queue, and collects. With--no-wait(launcher.wait=false) it submits and exits, and a small job that runs once the array has ended (--dependency=afterany) collects the batch; should that job not be submitted,scenariosays so, with the array's job id, and exits non-zero (scenario-slurm waitcollects the batch instead).
Each task, on its node:
- takes a slot of ports no other task on the node holds and nothing listens on (Ports);
- composes each of its cases' configs from the manifest's scenario and overrides, and refuses to run one that no longer resolves to the config it was submitted with (a job that does not run in the submitting environment);
- builds them (not checked again: the submitting process did), starts a
CARLA on its ports and runs its cases one after another through the same
queue a local batch uses, writing their files into the batch's directory
under the names a local batch gives them -- each Autoware stack's log
named after its case's stem (
autoware-<stem>.log), as a local batch names it, and a containerised backend's log after the task (backend_<name>-<array job id>_<task>.log), so tasks never append to one file (#2247). What a local batch decides once for all its cases, every task decides the same, however the cases are split into tasks (cases_per_task): the queue's settings (server, traffic,t4,t4scene, world reset, backends) are the batch's first case's, Rerun recordings are made when any case setsrerun.enabled(strict when any setsrerun.strict), and the canvas and the heavy metrics are the batch's. The manifest records which cases decide them (batch_settings), and a task composes those beside its own; - records each case's outcome as soon as it has run
(
slurm/results/<case>.json), and its own progress inslurm/tasks/<submission>-<task>.json(host, ports, state, peak resident memory); withrerun.enabled=trueit also converts each case's trajectory log into its Rerun recording ({stem}.rrd) and records whether that failed, socollectreports the recordings as a local batch does (Seeing a run). The submitting process only checks that thererunextra is installed; the nodes need it too; - stops its CARLA however it ends: at the end, on
scancel, on the signal SLURM sendslauncher.signal_grace_secondsbeforelauncher.timeruns out, and -- through the job script -- when the worker itself was killed, even by theSIGKILLthat ends ascancelafter SLURM'sKillWait(CARLA runs in a session of its own, which that signal misses): a small watchdog process of its own, started once the slot is taken, stops CARLA when the worker ends without having done so, and keeps the slot locked until then.
Following, collecting, resubmitting¶
scenario-slurm takes the batch's output directory:
scenario-slurm status outputs/2026-10-18/12-00-00 # cases, jobs, tasks
scenario-slurm wait outputs/2026-10-18/12-00-00 # follow, then collect
scenario-slurm collect outputs/2026-10-18/12-00-00 # summary, batch_results.json
scenario-slurm resubmit outputs/2026-10-18/12-00-00 # the cases with no result
scenario-slurm cancel outputs/2026-10-18/12-00-00 # scancel the batch's jobs
cancel gives scancel 60 s; one that has not returned by then is
stopped, cancel says so on one line and exits 1. The same 120 s limit on
sbatch, and the kept record, as above hold for resubmit; should the
collect job's sbatch time out, the array runs all the same and scenario
says how to collect it by hand.
status, collect and cancel take nothing but the batch directory. The
options of the others (scenario-slurm <command> --help says the same):
| Command | Option | |
|---|---|---|
wait |
--submission K |
the submission to follow, by its index (slurm/submissions/<K>.json; 0 is the first); default: the latest |
wait |
--poll-seconds S |
seconds between two polls of squeue; default: the batch's launcher.poll_seconds. Below 0.1 is taken as 0.1, a negative number is refused |
resubmit |
--failed |
also the cases that ran and failed |
resubmit |
--cases a,b |
these cases instead, by label or index; a case named twice runs once |
resubmit |
--array-parallelism K |
at most this many tasks at once (at least 1); default: the batch's launcher.array_parallelism |
resubmit |
--cases-per-task N |
cases a task runs (at least 1); default: the batch's launcher.cases_per_task |
resubmit |
--wait, --no-wait |
follow the new submission until it ends and collect, or submit and exit; default: the batch's launcher.wait. With nothing to resubmit, it collects at once, but only with an explicit --wait (and not with --dry-run) |
resubmit |
--force |
submit even while a job of the batch is queued or running, squeue cannot tell whether one is, or another submission of the batch is in flight or its sbatch timed out less than 600 s after its record was made |
resubmit |
--dry-run |
print the cases it would resubmit, submitting and collecting nothing, even with --wait, and exit 0; it still exits 1 where a resubmit would refuse: a busy batch without --force, an unknown --cases entry or an out-of-range option, and on an error reading the batch |
A task that raised, every failed attempt of a case and a native crash in a
task leave a crash report in the batch's crash_reports/,
as a local batch's do. A case's report runs the case again on its own, with
scenario and without launcher=slurm; a failed task's status names its
report (crash_report).
A case that has no outcome -- its task was cancelled, ran out of time, was
killed for memory, or its node failed -- is reported as ERROR with a message
saying so, and collect lists them, saying what to do about them, chosen from
the batch's records alone (#1803): where the case's task may still be queued
or running (it has a job, and it has no status yet, or its status says
starting, waiting, building or running), to wait for it with
scenario-slurm wait (with --submission <k> when its submission is not the
latest) if squeue --me still lists it and to resubmit once it has ended;
where its submission is in flight, to try again once sbatch returns; where its
sbatch timed out less than 600 s ago, to collect once the job ends if
squeue --me lists it, and otherwise to resubmit, which is refused until
those 600 s have passed unless with --force; and otherwise to resubmit. So
is a case whose recorded outcome holds no verdict -- no result object, or a
passed that is not true or false ("false" and 1 are none) -- or a
result that cannot be reported: a message that is missing or not a string,
an elapsed_seconds that is not a number (true is none),
condition_statuses that are not a list of objects each with a string label
and message and a boolean satisfied, or a failure that is not an object
or cannot be read (an at of the wrong type, say). What the worker writes
always is readable; rerun_recording and failure may be missing. It did not
pass, and collect reports it as ERROR naming its file; a rerun_error the
task recorded for it is kept in batch_results.json. status, wait and the
viewer count it as not passed: the results pages show it as an error.
An outcome is stale when the last submission that gave the case was to run
it again and its task stopped (failed or cancelled) before it reached the
case: the task removes a case's outcome only right before it runs the case,
so the one there names an earlier submission. collect reports it as ERROR
saying so, and status and the viewer count it as without a result (#2247).
resubmit submits again the cases with no outcome, no verdict or a stale
one and those that raised on every attempt (ERROR); --failed adds the ones that ran and
failed, --cases a,b names them (labels or indices; a case named twice runs
once), and --array-parallelism/--cases-per-task change how they are
spread. A case run again replaces its earlier files: before it runs, the task
removes every output of its stem (run_outputs), its T4 dataset directory
({stem}.t4) included, so none of the last attempt's is left beside the new
result. So resubmit refuses while a job of the batch is queued or running,
while squeue cannot tell whether one is, or while another submission of the
batch is in flight (--force submits all the same).
A batch directory that scenario-slurm cannot read or write -- a manifest it
may not read, or shared storage mounted read-only -- is one Error: ... line
naming the file, and the command exits 1, as for a directory that holds no
batch. collect (and wait, and resubmit --wait, which end in it) prints
the summary's verdict even when batch_results.json cannot be written, and
writes slurm/collected.json after its hints, so the verdict is not lost
(#1740).
Two resubmits of one batch may run at once -- from two viewer tabs, or from
the viewer and scenario-slurm resubmit. Each submission's record,
slurm/submissions/<k>.json, is created exclusively before sbatch runs,
which reserves its index k: a submitter that finds k taken takes the next
one, so no record is written over and no job id lost. The record is
hard-linked into place from a temporary file (O_CREAT | O_EXCL where the file
system has no hard links), which is atomic on NFS too, where flock may not
be; a reader never sees it empty. Until sbatch returns, its job_id is
null, and the submission counts as in flight: having reserved its index, a
resubmit checks the submissions before it once more, and refuses should one
be in flight or queued, so of two resubmits at once the later one submits
nothing. A record whose sbatch fails is removed. One left with no job id by a
submitter that was killed stops counting as in flight 600 s
(SBATCH_IN_FLIGHT_SECONDS) after it was made. wait follows a
submission only once it has a job id: on one still in flight it asks squeue
nothing, collects nothing and exits non-zero, saying to try again once sbatch
has returned; on one whose sbatch timed out, that the job may exist (check
squeue --me, then collect or resubmit); and on one left with no job id
it says the record is stale. status asks squeue nothing about a
submission with no job id either: it never reads as a job that ended, but as
no job id yet (sbatch in flight), as one whose sbatch timed out (above), or
as a stale record; the viewer shows its cases pending, unknown or missing
respectively.
A record that cannot be read
(empty, or not a JSON object) refuses a resubmit for as long after the file
was last modified, then is ignored with a warning. The other readers skip
such a record with a warning naming it: status lists it as unreadable,
cancel says it skipped it (cancel its job with scancel if it had one),
collect counts its cases as not submitted by it, and the viewer's batch page
shows the batch with a note on that submission.
collect can be run any number of times.
sacct is not needed: the batch is followed through squeue and the files the
tasks write.
The result viewer does the same from its pages: its run panel submits a
batch with launcher=slurm, and a SLURM batch's page shows its cases, jobs
and tasks as they run and resubmits, collects or cancels it
(Running through SLURM).
Configuration¶
launcher=slurm selects conf/launcher/slurm.yaml; every key can be set on the
command line (launcher.mem=12G), or in a config.
| Key | Default | |
|---|---|---|
partition, account, qos |
null |
sbatch --partition/--account/--qos; partition may list several (gpu,cpu) |
time |
null |
sbatch --time in any of its forms (90, 120:00, 168:00:00, 2-00, 1-12:30, 1-00:00:00, UNLIMITED); the task is sent SIGTERM signal_grace_seconds before it runs out |
mem, cpus_per_task, gres |
null |
sbatch --mem/--cpus-per-task/--gres (gpu:1, or shard:1 to share a GPU; a list: gpu:1,shard:2) |
array_parallelism |
null |
at most this many tasks at once (--array=...%K); null: no cap |
cases_per_task |
null |
cases a task runs on its CARLA; null: the cases spread evenly over array_parallelism tasks (one case a task without a cap), in no more tasks than the cluster's MaxArraySize (Array size) |
job_name |
null |
acs-<date>-<time> of the output directory |
extra_sbatch_args |
[] |
further sbatch arguments; --clusters/--cluster/-M is refused (see below) |
export |
null |
sbatch --export; null: ALL, or NONE with remote_cwd |
env_passthrough |
null |
variables recorded at submission and set again in the job (a matched name no shell takes, such as an exported bash function, is left out); null: see below, or none with remote_cwd |
setup |
[] |
shell lines the job runs before the worker (module load, source .venv/bin/activate); each is one command, and one ending in an unescaped backslash is refused |
env |
{} |
variables set in the job, last; a name must be a shell variable name ([A-Za-z_][A-Za-z0-9_]*), as the run panel requires |
python |
null |
the interpreter the jobs run (the tasks and the collect job): an absolute path, a path relative to the job's working directory (.venv/bin/python), ~/... (the home on the node), or a command on the node's PATH (python3); null: the submitting one |
remote_cwd |
null |
where the jobs work on the nodes, for a submit host that sees their file system at another path (A submit host with other paths); null: the directory scenario runs in |
headless |
true |
unset DISPLAY in the job, so CARLA renders off-screen |
start_stagger_seconds |
0 |
no task of the batch starts sooner than this after another did: the first wave, the tasks started as slots free up, a resubmit's (Shared state) |
wait |
true |
follow and collect; false (or --no-wait): submit and exit |
poll_seconds |
15 |
how often wait asks squeue |
cancel_on_interrupt |
true |
Ctrl-C while waiting cancels the batch's jobs |
collect_job |
true |
with wait: false, a job that collects once the array ends |
collect_mem |
2G |
its --mem |
signal_grace_seconds |
120 |
--signal=B:TERM@N; at least 0 |
port_base, port_slots |
20000, 1000 |
where the tasks' port slots are (Ports); port_base at least 1024, port_slots at least 1, and every slot's ports within 65535 |
port_lock_dir |
null |
where the slots' locks are; null: /dev/shm/acs-slurm-ports-<uid> (per user, so slots are exclusive among your own tasks only), else /tmp/acs-slurm-ports-<uid>, each used only if private; users sharing nodes set one group directory (Ports) |
Each value is checked before anything is written, and one of the wrong type is
refused with an error naming its key (launcher.<key>), never quietly read
as something else:
- The counts (
cpus_per_task,array_parallelism,cases_per_task),signal_grace_seconds,port_baseandport_slotsare whole numbers:2,2.0or"2", not2.7,true, NaN or infinity. The counts andport_slotsare at least 1. - The seconds (
start_stagger_seconds,poll_seconds) are finite numbers, at least 0, nottrue. - The flags (
wait,headless,collect_job,cancel_on_interrupt) aretrueorfalse, or one of their spellings as text ("false","no","off","0", and"true","yes","on","1");"false"is false. - The text keys are text or a number, not a boolean, a list or a mapping;
extra_sbatch_args,env_passthroughandsetupare lists (or one string) of non-blank text, without a null or blank item;envis a mapping (or null, no variables) whose values are text or a number, not null or a boolean (quote one:FLAG: "true"). extra_sbatch_argsmay not send the batch to another cluster (--clusters,--cluster,-M, in any spelling:--clusters=x,-Mx):squeue,scanceland the collect job's dependency ask the default cluster, so following,resubmit's check that the batch is not running, andcancelwould miss it. Shouldsbatchreport a cluster all the same (SBATCH_CLUSTERSin the environment), the submission fails, naming the job toscancel -M(#2247).0,falseor[]is a value, checked as any other, never read as unset;launcher.typeislocalonly when it is missing or null ("",falseor0is refused).
cases_per_task trades a task's start -- CARLA takes about 50 s to start, and
a driver policy about as long to load -- against how evenly the cases spread
and how many a lost task takes with it.
Array size¶
SLURM caps an array's size: its task ids run from 0 to one less than the
cluster's MaxArraySize (scontrol show config; 1001 by default), so an
array holds at most MaxArraySize tasks. Without cases_per_task the launcher
reads MaxArraySize from scontrol show config (1001 when scontrol cannot
be asked or names none) and puts as many cases in a task as it takes to stay
within it: 2500 cases on the default limit run in 834 tasks of up to 3 cases.
A cases_per_task you set is taken as it is; should it make more tasks than
MaxArraySize, nothing is submitted and the error says the cases_per_task
that fits. Should sbatch still refuse the array (a limit scontrol did not
report), its error says to set launcher.cases_per_task.
The job's environment¶
A task runs with the Python the batch was submitted from (sys.executable,
or launcher.python) and its environment:
sbatch --export=ALL(launcher.export) hands the job the environment of the shellsbatchruns from;- the job script then sets again, as they were when the batch was submitted,
the variables
launcher.env_passthroughmatches:PATH,LD_LIBRARY_PATH,PYTHONPATH,PYTHONHOME,VIRTUAL_ENV,CONDA_PREFIX,CARLA_*(CARLA_EXECUTABLE),AUTOWARE_*,ONEPLANNER_*,TYPESAFE_CARLA_*,GLIBC_TUNABLES,MALLOC_*,CUDA_HOME,CUDA_PATH,CUDAToolkit_ROOT-- soscenario-slurm resubmitfrom another shell runs the cases as they were submitted; launcher.setupruns;launcher.envis set.
The collect job (launcher.collect_job) runs its script,
slurm/jobs/<submission>-collect.sh, with the same environment: steps 1 to 4,
then launcher.python in the directory the batch was submitted from, so a
setup that loads a module or activates a venv serves it as it does the tasks.
A driver policy's variables belong on env_passthrough (a pattern:
launcher.env_passthrough=[...] replaces the list) or in launcher.env. No
token or credential is on the default list, and a name with TOKEN,
SECRET, KEY, PASSWORD or CREDENTIAL in it (in any case) is not
recorded through a pattern (AUTOWARE_*) either -- only when
env_passthrough names it exactly, or launcher.env sets it: the manifest
and the job scripts holding the values are files in the batch's output
directory (both the user's alone, mode 0600 and 0700). With
export: NONE, the job has only what steps 2 to 4 give it.
With launcher.remote_cwd set, none of this comes from the submit host: the
job has the environment SLURM gives it on the node (export: NONE), the
variables env_passthrough names only when it is given, then setup and
env (A submit host with other paths).
A config that reads the environment (${oc.env:ONEPLANNER_MAP_DIR}) is
resolved in the task too; a case whose config resolves differently there than
where it was submitted is refused, naming the keys that differ.
A submit host with other paths¶
By default the jobs run where scenario was run: the directory it ran in, the
interpreter it ran with and the batch's output directory are recorded by
their absolute paths, which the nodes must see as the submit host does. A
submit host that sees the nodes' file system somewhere else -- a workstation
that mounts a node's ~/workspace/OnePlanner with sshfs or NFS at
~/mnt/node/OnePlanner, under another user's home -- names the nodes' paths
instead, none of them absolute:
# On the submit host, in its mount of the node's ~/workspace/OnePlanner:
cd ~/mnt/node/OnePlanner
uv run scenario --multirun 'scenario=cut_in/*' +sweep.odd_sample.count=24 \
launcher=slurm launcher.partition=main launcher.gres=shard:1 \
launcher.remote_cwd=workspace/OnePlanner \
launcher.python=.venv/bin/python \
"launcher.setup=['source ~/workspace/opt/env.sh']"
launcher.remote_cwdis where every job -- each task and the collect job -- works on the node. A relative path is relative to the job user's home on the node,~/...is that home too, and an absolute path is used as it is. It is expanded by the job script on the node, never here: the home is the one the node's passwd database gives the job's user (whatever$HOMEthe job was handed), so the user name and home may differ from the submit host's.launcher.pythonis then required: the submit host's interpreter is not the node's. A relative path (.venv/bin/python) is relative toremote_cwd, a bare name (python3) is looked up on the node'sPATHassetupleaves it.- The batch's output directory must lie under the directory
scenarioruns in (outputs/<date>/<time>/does), which is what the node sees atremote_cwd;scenariorefuses a batch elsewhere. The manifest records it relative to that directory (batch_dir_relative, a version 2 manifest), and the jobs find it under their own working directory. On the submit host,scenario-slurmtakes the batch's directory as this host sees it (scenario-slurm wait ~/mnt/node/OnePlanner/outputs/<date>/<time>), andresubmitsubmits with the settings the batch was submitted with. - Nothing of the submit host's environment reaches the job:
exportisNONE(the job starts with the environment SLURM gives it on the node, which on most clusters is the user's login environment) andenv_passthroughis empty, unless either is set. Set the node's environment withsetup, which runs inremote_cwdand may read unset variables (source ~/workspace/opt/env.sh,module load cuda,source .venv/bin/activate), andenv. - SLURM writes a job's output before the job knows where the batch is: each
job starts in
/tmpwith its output there (/tmp/acs-slurm-<user>-<job>_<task>.outon the node) and sends it on to the batch'sslurm/logs/once it is inremote_cwd. The file in/tmpkeeps what came before (aremote_cwdthe node does not have) and what SLURM itself writes (a time limit, a cancel). - A case's config that resolves to absolute paths -- an ODD file in a scenario package, a map -- resolves to the node's paths there; see below for when that is still the same config.
Without remote_cwd the batch is submitted and run exactly as before (a
version 1 manifest).
A value that is a path may differ and still be the same config. That is the
case when it is the submitting directory's path seen from the job's working
directory (<cwd>/x there, <remote_cwd>/x here), or when it names a file
with the content it had where the batch was submitted: scenario records the
SHA-256 of every file an absolute path in a case's config names (the
manifest's files), and the task compares the file its own config names. A
file missing on the node, or with other content, is refused, naming the key
and both paths. This holds with or without remote_cwd: a package installed
in another environment names its files at other paths too.
Ports¶
Each task's CARLA needs ports no other CARLA on the node uses, and a user may
run one on the defaults (2000, TrafficManager 8100). A task takes a slot of 10
ports from port_base:
| Port | |
|---|---|
base+0 |
CARLA RPC (server.port) |
base+1 |
CARLA streaming (-carla-streaming-port) |
base+2 |
CARLA secondary (-carla-secondary-port) |
base+3 |
TrafficManager (traffic_manager.port, and the traffic backend's own: traffic.options.port, sumo's traffic.options.tm_port) |
base+4 |
the Autoware bridge the run hosts (autoware.address) |
base+5 |
the driver bridge the run hosts (driver.bridge.address) |
The slot a task tries first is its array job id plus its task id, so the tasks
of an array start on different slots. It takes one only when no other task
using the same lock directory holds it (an exclusive lock,
<lock dir>/autoware_carla_scenario-port-<base>.lock, which goes when the
task's process does) and none of its ports is bound; else the next. The lock
directory must be one every job on the node sees, which /tmp is not where
SLURM gives each job a private one (job_container/tmpfs): it is
/dev/shm/acs-slurm-ports-<uid> (or /tmp/acs-slurm-ports-<uid> where that
cannot be made), or launcher.port_lock_dir. The default range, 20000-29999, is below Linux's
ephemeral ports.
The default lock directory is per user, so slots and ROS domains
are exclusive among your own tasks only. Another user's
task on the same node may take the same slot: the check that none of its ports is bound stops
that only once the other task's CARLA has bound them, tens of seconds after
it took the slot. Users who share nodes set launcher.port_lock_dir to one
directory they all write to, for example a group's own:
mkdir -p /dev/shm/acs-slurm-ports-mygroup
chgrp mygroup /dev/shm/acs-slurm-ports-mygroup
chmod 2770 /dev/shm/acs-slurm-ports-mygroup # group-only, setgid, no sticky bit
The lock files are created 0644 less your umask, so keep a umask that
leaves them readable by the group (022, 002 or 027). Do not use a
world-writable directory: anyone on the node could then create or hold the
lock files and keep your tasks off every slot. That is also why the default
is not shared.
For the same reason a default directory is used only if it is private: a
real directory (not a symlink) that you own and that neither your group nor
others can write to. One another user made first, or one open to writing, is
passed over with a warning for /tmp/acs-slurm-ports-<uid>, checked the same
way; if that is not private either, the task stops with an error naming both.
Have the node's administrator remove a directory another user made (chmod
go-w one of your own), or set launcher.port_lock_dir. A
configured port_lock_dir is taken as it is, as it may be shared on purpose.
Lock files are never opened through a symlink.
The TrafficManager's port is always the slot's, whatever the config says: the traffic
groups interpolate the backend's own (traffic.options.port, sumo's
traffic.options.tm_port) from traffic_manager.port, and one set outright
(traffic.options.port=8101, or null for the backend's default) is
replaced by the slot's too, with a warning naming the key; otherwise every
task on the node would ask its CARLA for a TrafficManager on that one port.
driver.address is not changed: a policy the run serves itself
(driver.policy) listens on a port the OS picks, and one it does not is a
server someone else runs, which every task dials. driver.transport is not
changed either: with direct a task calls the policy it serves without a
socket, and with unix it serves it on a socket in a directory of its own
(under $TMPDIR, which SLURM does not set by default, so usually /tmp), so neither needs a port of the slot and tasks
sharing a node never meet; a unix: driver.address someone else serves is
dialled by every task, as a TCP one is.
ROS domains¶
The ROS 2 nodes of tasks on one node would share a DDS domain: they all start
from the same config. So each task maps every domain id its config names to
one no other task on the node holds (slurm/ports.py, RosDomains), and
keeps it for all its cases:
| Config | What the task changes |
|---|---|
ego.entity=ros2 |
ros2.domain_id |
autoware.launcher.type=docker |
each autoware.launcher.ros_domain_ids entry; with none, one taken from autoware.launcher.env.ROS_DOMAIN_ID, else 0 |
autoware.launcher.type=command |
ROS_DOMAIN_ID in autoware.launcher.env (from its own value, else the job's exported ROS_DOMAIN_ID, else 0) |
An id maps to the first domain from it on (up to 232, then from 0) whose lock,
<lock dir>/autoware_carla_scenario-ros-domain-<id>.lock beside the port
slots', no other task holds; the lock goes when the task's process does. The
first task on a node gets the ids it asked for, the next ones the ids after:
with ros_domain_ids=[10,11], one task runs on 10 and 11 and the next on 12
and 13. The ids are checked as a local run checks them (whole numbers from 0 to 232;
a string ros_domain_ids is refused), naming the key.
A task whose domain is not the one configured logs a warning naming both and
records the mapping in its progress file (slurm/tasks/<submission>-<task>.json,
ros_domains, configured id to domain, beside ports). With
ego.entity=ros2 that matters: ROS 2 nodes started outside the framework
must join the task's domain (ROS_DOMAIN_ID=<domain>), not the configured
one, so read it there, or give each such task a node to itself.
The domains are not part of the port slot: a slot's index runs to
port_slots (1000 by default), and ROS 2 has 233 domains. Keep
array_parallelism times the domains a task uses well below that, and below
102 on Linux (0 to 101 is what the ROS 2 documentation calls safe). The locks
are per user (the lock directory is), so another user's tasks on the node
may still pick the same domain, and nothing checks whether a domain is in
use: users who share nodes share a lock directory (Ports).
Autoware stacks¶
The docker launcher labels each container it starts with your uid and with
its owner, autoware_carla_scenario.owner=<host>:<pid>. A task removes only
the leftover containers whose owner no longer runs, so a live stack of
another task on the node is left alone (autoware.md). Tasks
that build one workspace (build: always, or auto while it is unbuilt) do
so one at a time, under a lock in the workspace: a task that waits for
another's auto build uses it rather than build again.
Limits¶
- Memory. A task runs a CARLA (about 6 GB resident) and a scenario process
(about 6 GB with a TensorRT driver policy loaded). Without cgroups, SLURM
uses
--memonly to schedule, so set it, andarray_parallelism, to what the node has. - Shared state. The tasks share the map caches (
~/autoware_data/maps), the map files a driver policy reads (driver.map_dir) and the batch's output directory, so these must be on a file system every node sees. Map files, and the OpenDRIVE a run captures from the loaded world on a map's first run, are written atomically, so tasks that fill a cache at once do not break it. A driver policy's own caches are its own: one that two processes cannot load at once (OnePlanner's TensorRT engines, whose identity file is written under one temporary name, so two loads at the same moment fail one case withfilesystem error: cannot rename) needsstart_stagger_seconds, enough for one task to load it before the next starts (30 s for OnePlanner, which loads it once a task). The stagger is held withflockonslurm/start.lock, or where the file system has none, with a lock file beside it created withO_EXCL. One a killed task left is taken over once it isstart_stagger_seconds+ 60 s old by the file server's clock, so a node whose clock is off does not break a live one. One waiter takes a stale lock over: the waiters share a mutex of.takeover.*files beside the lock, one per attempt, created withO_EXCL(a waiter killed holding one is superseded by the next attempt 10 s later), and the one that wins judges the lock stale again before removing it. A task removes the lock on release only while it is still its own. So no waiter removes a live lock, and no two tasks start together, except when a task is still alive past the point its file is judged stale: one that held the lock that long, or a waiter that stalled for 10 s in the middle of taking a lock over. - ROS 2 and Autoware egos (
ego.entity=ros2,ego.entity=autowarewith a launcher) run their stacks on DDS. Each task takes ROS domains of its own (ROS domains), and its Autoware containers are told apart from another task's (Autoware stacks), so tasks sharing a node do not meet. A command launcher's script that setsROS_DOMAIN_IDitself rather than reading it from its environment is the exception. - File names. A case is named by its position in the batch, which is the name a local batch gives it whenever every earlier case opened its trajectory log. A case that raises before it opens it (while its ego is created or the world is set to synchronous mode) leaves its name to the next case in a local batch; under SLURM every case keeps its own.
build_scenario_fn. A batch run from code with abuild_scenario_fnis run by the tasks only when that function can be imported by name.