Backends a Run Starts¶
Some parts of a scenario run on more than CARLA. One example is a pedestrian
whose motion a model generates (SmartPedestrianControllerAction, which uses
the ARDY container). Another is a renderer served by a process of its own. Such
a part declares on its class the backends it needs, with @need_prepare:
from autoware_carla_scenario.prepare.declare import need_prepare
from autoware_carla_scenario.pedestrian.ardy import ArdyBackend
@need_prepare(ArdyBackend)
class SmartPedestrianControllerAction(BaseAction): ...
The scenario queue then starts what its scenarios need before they run, and stops it when the queue stops.
When a backend starts¶
The queue has a preparer, prepare.pool.BackendPool, which scenario
builds from backends. The preparer looks at each scenario before it runs:
- When the queue starts, before the CARLA server, it starts what the queue's scenarios declare.
- The scenario class's own
@need_preparecounts. - So do the declarations on the classes that a declarative (YAML) scenario's actions build. These are read from the compiled document without CARLA.
- On SLURM, the worker does the same for its task's cases once its CARLA is up.
- After each scenario's
setup(), before its run ticks, it starts whatever the registered actions need that is not running yet. This covers a Python scenario whose actions only exist oncesetup()has built them.
Each backend starts at most once per queue. The next scenario that needs it
finds it running. An action reaches the backend it needs with
prepare.pool.running_backend(ArdyBackend).
An action that runs without a queue starts its backend itself the first time it fires, with that backend's defaults, and stops it when the process exits.
Resources¶
Before starting anything, the preparer adds up what the backends it is about to
start take, as each one declares in prepare.Resources:
- GPU memory on each GPU it runs on, in MiB;
- host memory, in MiB.
It then compares that with what the host has free:
- the free memory of each GPU this process may use, from
nvidia-smi, limited byCUDA_VISIBLE_DEVICES(SLURM sets it for a job's GPUs); MemAvailable, the host's own, since a container runs outside the job that starts it.
If anything falls short, the run fails with InsufficientResourcesError before
any backend starts. The message names what each backend needs and what is
free:
InsufficientResourcesError: the host cannot run the backends this run needs --
ardy: 24000 MiB of GPU memory needed, 9120 MiB free on the roomiest GPU (0)
(needed: ardy (24000 MiB GPU, 16000 MiB host)). Free the memory, or lower what
a backend is said to take (backends.<name>.gpu_memory_mb / memory_mb).
The GPU memory is checked on the GPUs docker run gives the container:
gpus: auto: the GPU the preparer places it on, and it is started with--gpus device=<that GPU>. Each of these, the largest first, is placed on the GPU with the most room left. When that leaves one without room, the other ways to place them are tried, and the run is refused only when none fits them all; the message then names the ones the first placement left out;gpus: device=N: GPUN;gpus: '"device=N,M"'(quoted, asdocker run --gpustakes a list): each GPU listed;- a count,
gpus: N: GPUs0toN-1, the onesdocker run --gpus Ngives; gpus: all: every GPU this process may use.
A container on GPUs it is set to (every case but auto) needs its
gpu_memory_mb free on each of them, and is placed before the auto ones. One
with gpus: none needs no GPU at all. A gpus that does not say which GPUs
docker gives (a GPU's UUID, other --gpus options) is refused for a backend
that takes GPU memory, naming backends.<name>.gpus and
backends.<name>.gpu_memory_mb: set gpus to one of the above, or
gpu_memory_mb: 0 to skip the check.
The check is a snapshot of what is free when the backends start. It cannot see what others are about to take:
- CARLA, when the queue starts it, starts after the backends. Its GPU memory must be left free, beyond what the backends take.
- Another run on the same host that has started a backend that is still loading has not taken its GPU memory yet. Two runs that prepare at the same time can both pass the check and run out of GPU memory when they load.
CUDA_VISIBLE_DEVICESis read asnvidia-smi's indices or UUIDs. Where SLURM renumbers a job's GPUs from 0, setgpus: device=Nto the host's index instead.CUDA_VISIBLE_DEVICESdoes not limit what docker gives a container: withgpus: allit still gets every GPU of the host, while only the visible ones are checked. A GPU thatdevice=or a count names and that is not visible is refused. Under SLURM, preferautoordevice=N.
Containers¶
prepare.docker.DockerBackend runs a backend as a container on this host. It:
- checks that docker is usable, and warns when it has no nvidia runtime;
- pulls the image if the host lacks it (
pull: missing, the default; alsoalwaysornever).docker runitself never pulls (--pull never), so withpull: neveran image the host lacks fails the start at once; - runs the container on its GPU, with its port published on
127.0.0.1at a free port; - follows the container's log into the run's output directory
(
backend_<name>.log;backend_<name>-<array job id>_<task>.login a SLURM array task, each task starting its own); - waits until the service answers, for up to
startup_timeout_s.
A container that exits while starting is kept until its logs are read into the
error, and then removed; every container is removed when its run ends. So is
one docker run created but failed to start (a port taken in the meantime, a
GPU error, a timeout, Ctrl-C), and one whose docker stop timed out
(docker rm --force).
It labels each container with the host and process that started it. A container left behind by a killed run is removed the next time a backend of its name starts. Only that one is removed: two runs on one host, such as two SLURM jobs, each keep their own.
Settings¶
Each backend reads its own block under backends of the config, and every key
is optional:
backends:
ardy:
image: ghcr.io/hakuturu583/carla_ardy_backend:0.1.0
model: soma # the ARDY checkpoint
acceleration: none # none, compile, trt
gpus: auto # the GPU the preparer chose; or all, device=N
hf_home: ~/.cache/huggingface
gpu_memory_mb: 24000
memory_mb: 16000
startup_timeout_s: 1800 # a first start downloads ~20 GB of weights
timeout_s: 120 # one generation step
pull: missing
docker_args: []
The settings are checked when the backend is built, before any image is pulled or container started. A key that is absent or null takes its default. The settings every container backend shares:
imageis a non-empty image name, andgpusauto,all,noneor whatdocker run --gpustakes (device=N, a count of 1 or more:gpus: 0is refused, usenone).autois the GPU the preparer placed the backend on, and it places one only when it asks for GPU memory: withgpu_memory_mb: 0there is no placement, andautopasses--gpus all, every GPU, asalldoes (nonegives it none). Withgpu_memory_mbabove 0,gpusmust say which GPUs docker gives the container (Resources);pullismissing,alwaysornever;startup_timeout_s(and ARDY'stimeout_s) is a finite number of seconds above 0;gpu_memory_mbandmemory_mbare whole numbers, not negative;docker_argsis a list ([--shm-size=8g]), never a string.
ARDY's own settings: model and hf_home are non-empty text, and
acceleration is none, compile or trt. A false, a number or '' is
refused rather than handed to the container as its text.
A value refused is an error naming its key, such as
backends.ardy.startup_timeout_s must be finite and above 0, got -1. A
run refuses the settings of a backend this package ships (backends.ardy)
before anything starts, along with the rest of its config. A backend of
another package has its settings checked when its scenario is prepared.
The ARDY text encoder is Llama-3-8B, which is gated on Hugging Face. The
container gets the host's Hugging Face cache, where the token and the weights
are, at hf_home. It also gets HF_TOKEN when this process has one.
Adding a backend¶
A backend class has two parts:
- a
backend_name, which is its settings' key underbackends; - a
from_settings(settings)class method that builds it.
What it builds implements prepare.Backend: name, resources(),
start(log_dir, gpu), address() and close(). For a container, subclass
prepare.docker.DockerBackend, build its DockerSettings with
prepare.docker.read_docker_settings(name, settings, ...), which reads and
checks the shared keys above (check your own keys with its helpers, such as
positive_seconds), and implement
ready(timeout_s) to say when the service answers. Then put
@need_prepare(YourBackend) on the action, scenario or renderer that needs it.