Packing a scenario package into a container image¶
A scenario package created with scenario-new is a normal Python
distribution, so it can be shipped as a container image that carries the
framework, the scenario, its configs, and its CARLA client --
typesafe_carla, whose released
wheel carries its CPython package prebuilt, so nothing is compiled at build
time or in a container.
The image is built by the pack-scenario-image composite action. Everything it
needs lives in the action's own directory, so it is meant to be referenced from
other repositories:
The ref is both the action version and the framework version that ends up in
the image — the runner checks the whole repository out alongside the action,
and the framework packages are built from that copy. Pin a release tag
(vX.Y.Z, one of the repository's
tags)
for reproducible framework versions; @master follows the latest framework
instead. Your own repository does not have to contain this framework, or even
be checked out.
What the image contains¶
Only installed code:
| Present | Absent |
|---|---|
/opt/venv — the framework, the scenario package, their dependencies (typesafe-carla, with its prebuilt CPython package, and its Codon toolchain among them) |
Source trees, pyproject.toml, tests, docs, git history |
uv, wheels, build caches, a C compiler, any apt package at all |
The entry point is the scenario CLI and the working directory is /work,
where Hydra writes its run directory.
/opt/venv arrives as four layers, not one, so a client that
already holds another image built the same way pulls only what actually
differs.
Stage layout¶
.github/actions/pack-scenario-image/Dockerfile runs three stages, each
handing the next strictly less than it was given:
flowchart LR
A["wheelhouse<br/>(Alpine + uv)<br/>framework/ + scenario/ → wheels"]
B["venv<br/>(Debian slim)<br/>four installs → /export/<layer>,<br/>then slimmed"]
C["runtime<br/>(Debian slim)<br/>four COPY --link layers,<br/>no apt layer"]
A -- "*.whl + requirements.txt" --> B
B -- "/export/* only" --> C
wheelhouse is Alpine because everything it touches is pure Python — uv
build emits py3-none-any wheels for the framework's workspace package and
for the scenario package — so the stage needs
neither glibc nor a compiler, and nothing but the wheels survives it.
The runtime is not Alpine, and cannot be. typesafe-carla and its Codon
toolchain, opencv-python-headless and simple-lanelet2 publish manylinux
wheels only. There is no musllinux wheel to fall back to, so a musl runtime
cannot install them at all. Debian slim is the smallest base that runs the
code.
Neither the venv stage nor the runtime compiles anything, and neither
installs a C compiler: the typesafe-carla wheel on PyPI carries typesafe_carla's
CPython package prebuilt (typesafe_carla/carla/_prebuilt, one build for every
Python 3.10+), so installing the wheel is all the client needs. typesafe_carla
falls back to compiling that package on its first import (15-50 minutes, ~14 GB
of memory, cc) only where no matching prebuilt one exists -- a typesafe-carla
installed from source, or a typesafe-carla-toolchain other than the one the
wheel was built with -- and the image rules that out: the build imports the
client from the finished virtualenv with TYPESAFE_CARLA_PYCARLA_BUILD=0, and
the runtime sets the same variable, so a prebuilt package that does not apply
is an ImportError, at build time or on the first run, rather than a
half-hour compile inside a container.
Layer layout¶
An image is pulled as a stack of layers, and a client downloads only the ones
it does not already have. A virtualenv shipped as a single COPY is a single
layer, so changing one line of a scenario means pulling all ~400 MB of it
again. The Dockerfile therefore installs /opt/venv in four steps and ships
each step as its own layer, ordered from the input that changes least often to
the one that changes on every commit:
| # | Layer | Holds | Rebuilt when |
|---|---|---|---|
| 1 | client |
The virtualenv, typesafe-carla with its prebuilt CPython package (import typesafe_carla.carla as carla) and its Codon toolchain (typesafe-carla-toolchain), at the versions uv.lock pins |
uv.lock moves typesafe-carla or its toolchain, or python-version or the base image changes |
| 2 | deps |
The rest of the framework's third-party dependency closure — OpenCV, scipy, numpy, the matplotlib stack, … | uv.lock, python-version or the base image changes |
| 3 | framework |
autoware-carla-scenario |
Framework code changes |
| 4 | scenario |
The scenario wheel, plus any dependency it adds of its own | The scenario package changes |
(Before the move to typesafe_carla the dependency layer was ~400 MB and the two framework and scenario layers ~2 MB and ~80 kB; the client layer, with the Codon toolchain in it, has not been measured in an image yet.)
The client comes first because it depends on nothing else: layer 1 installs typesafe-carla alone, constrained to the lock's versions, so its content is a function of the typesafe-carla and typesafe-carla-toolchain versions and the base image and nothing more. Two images built for the same client share that layer whatever else they contain, and a lock change that leaves the client alone moves only the dependency layer behind it.
So editing a scenario and rebuilding transfers the fourth layer. Bumping the framework transfers the third and fourth. A lock change moves the dependency layer, and the client layer only when it moves typesafe-carla or its toolchain. The push is incremental for the same reason: a registry that already holds a blob is sent the new layer and nothing else.
What makes it hold¶
Layer reuse is by digest, and a digest covers file metadata as well as content, so two things could quietly defeat the split:
- Install timestamps. The same wheel installed twice produces the same
bytes at different times, and that alone is a different layer.
venv-layer.shtherefore stamps every exported file with one fixed timestamp (LAYER_MTIME, 2020-01-01 by default) as it captures it, and re-stamps afterwards whatever the slim pass rewrote. A rebuild from a cold cache reproduces all four layers byte for byte: every one of them, the client included, is installed from wheels, and nothing in the image is compiled. - The layers underneath. A layer can only be reused together with its whole
parent chain, so anything that varies has to sit behind the virtualenv
rather than in front of it. The runtime stage therefore copies the four
layers first and only then creates the user:
useraddwrites the current date into/etc/shadow, which would otherwise put a fresh digest on every layer behind it. The copies useCOPY --link, so each layer is built against an empty base and stays the same blob wherever it lands.
slim-venv.py runs once per layer rather than once over the virtualenv, so
stripping a shared object rewrites it inside the layer that installed it
instead of duplicating a stripped copy into a later one. The build then
reassembles the four trees exactly as the runtime stage stacks them and imports
the native extension modules from the result, so a file lost or shadowed by the
split fails the build rather than the container.
Sharing the build cache¶
Layers are about the pull; cache-scope is the same idea for the build. It
defaults to one scope per image, but images that share a lock and framework
are the same build up to their scenario layer, so pointing several of them at
one scope lets the later ones restore the client and dependency stages instead
of resolving and installing them again. The action reports the scope
it used, so a second build in the same workflow can just name the first's:
- uses: ./.github/actions/pack-scenario-image
id: first
with:
scenario-package-path: packages/first_scenario
image: ghcr.io/my-org/first-scenario
- uses: ./.github/actions/pack-scenario-image
with:
scenario-package-path: packages/second_scenario
image: ghcr.io/my-org/second-scenario
cache-scope: ${{ steps.first.outputs.cache-scope }}
Keeping the image small¶
Two things do the work, measured on a scaffolded package with the official CARLA 0.10.0 client (before the move to typesafe_carla, whose Codon toolchain adds to these totals):
| virtualenv | |
|---|---|
| Naive install | 565 MB |
opencv-python-headless instead of opencv-python |
512 MB |
…plus the slim pass |
413 MB |
- Headless OpenCV. The framework's only OpenCV calls are
imencodeandpointPolygonTest; there is noimshowornamedWindowanywhere, so the GUI build's GTK/GStreamer payload is dead weight. Dropping it also removes the last external shared-library dependency: every.soin the venv now needs nothing beyond glibc,libgcc_s,libstdc++andlibz, all of which the base image already has. The runtime therefore installs no apt packages and carries no apt layer (thewith-ffmpeginput went with the video in 15.0.0). -
The
slimpass (on by default,slim: "false"to disable) drops byte-code caches, vendored test suites, C headers, Cython sources and type stubs (about 26 MB), then runsstrip --strip-unneededover every bundled shared object (64 MB) — the manylinux wheels for scipy, numpy and OpenCV ship unstripped. Stripping leaves the dynamic symbol table intact. The stubs of mypy's bundled typeshed (mypy/typeshed/, about 3 MB) are kept: the image runs mypy for the scenario check's None check, and without the standard library's stubs mypy stops with "Failed to find builtin module 'typing', perhaps typeshed is broken?", which the check reports only as skipped. The build checks a small program with mypy and fails if mypy does not report its error.GNU binutils up to 2.40 — the version in Debian bookworm — rewrites some objects into a layout the kernel's ELF loader rejects with "ELF load command address/offset not page-aligned"; scipy's bundled OpenBLAS is one. So
slim-venv.pyre-reads every object it strips and restores the original unless eachPT_LOADsegment still satisfies the loader's congruence rule. A newer binutils strips everything and restores nothing; on bookworm the one restored file costs about 2 MB. The build then imports the native extension modules and fails right there if anything did break. - Stage separation. Wheels are built in a throwaway Alpine stage and the virtualenv in a throwaway Debian stage, so no source tree, wheel, build cache or copy ofuvreaches the final image.
The layer layout does not change any of these totals — the same bytes are simply split across four layers instead of one, so that a rebuild transfers only the layers that changed.
What is left is dominated by dependencies with no smaller variant: OpenCV,
scipy, numpy, and the matplotlib stack that pyxodr requires.
Pinning the CARLA client — and everything else¶
The client is the typesafe-carla that uv.lock pins (CARLA UE5: the wheel's
LibCarla is built from a CARLA ref, which the build prints), with the CPython
package that wheel carries prebuilt, loaded before the image is finished. Nothing in the image
can change it afterwards. The default tags are :latest and the commit's short
SHA (:<short sha>, its first 7 characters); to see which client an image
carries:
docker run --rm --entrypoint python ghcr.io/tier4/my-scenario:latest -c \
"from importlib.metadata import version; from typesafe_carla import paths; print(version('typesafe-carla'), paths.native_info()['carla_git_ref'])"
The whole dependency tree is pinned, so the same ref rebuilt months
apart gives the same image: the build context carries uv.lock, the wheelhouse
stage exports it as a requirements file with uv export --frozen, and that
file is what constrains the client install, what the dependency layer
installs, and what constrains the two installs after it. This buys reproducibility rather than size — and, with the
timestamps normalised, it is what lets an unchanged dependency layer keep its
digest across rebuilds. Constraints only bound what is resolved, so a scenario
package that brings dependencies of its own still installs — they are the one
part that floats, and they land in the scenario layer.
In a workflow¶
jobs:
pack:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Pack the scenario package
uses: ./.github/actions/pack-scenario-image
with:
scenario-package-path: packages/my_scenario_package
image: ghcr.io/${{ github.repository_owner }}/my-scenario
push: "true"
The scenario package does not have to live in this repository — check it out
anywhere on the runner and point scenario-package-path at it.
Inputs¶
action.yml carries the full list with defaults; these are the ones worth
explaining further:
| Input | Meaning |
|---|---|
scenario-package-path |
The generated package — the directory with its pyproject.toml. Required, and may sit outside this repository. |
framework-path |
uv workspace root supplying the framework. Defaults to the action's own checkout, which is what makes the pinned ref the framework version. Its members are read from [tool.uv.workspace] members, globs included. |
slim |
Strips the virtualenv (see above). On by default. |
platforms |
Target platforms. Empty (the default) builds for the runner's own: linux/amd64 on an x86_64 runner, linux/arm64 on an arm64 one such as ubuntu-24.04-arm — see Architectures. |
cache-scope |
Cache key namespace. Defaults to one scope per image and platform, so images built in the same workflow do not evict each other — give two images the same scope to share it instead. A given scope is used as is, so give each platform its own. |
Outputs: image-ref, tags, digest (pushed images only), cache-scope.
Architectures¶
typesafe-carla and its Codon toolchain ship wheels for Linux x86_64 and aarch64, and so does every other native dependency of the image, so the action packs for either. Run it on an arm64 runner to get an arm64 image:
jobs:
pack:
strategy:
matrix:
include:
- { runner: ubuntu-latest, arch: amd64 }
- { runner: ubuntu-24.04-arm, arch: arm64 }
runs-on: ${{ matrix.runner }}
steps:
# ...
- uses: ./.github/actions/pack-scenario-image
with:
scenario-package-path: packages/my_scenario_package
image: ghcr.io/${{ github.repository_owner }}/my-scenario
# One tag per architecture: the default tags are the same on both
# legs, so whichever pushed last would win.
tags: ghcr.io/${{ github.repository_owner }}/my-scenario:${{ github.sha }}-${{ matrix.arch }}
push: "true"
Each leg builds, loads and smoke-tests its own image natively. To publish one
multi-architecture tag, join the per-architecture tags in a later job with
docker buildx imagetools create. Cross-building on one runner
(platforms: linux/amd64,linux/arm64 after docker/setup-qemu-action) also
works, but emulated. A multi-platform build cannot be loaded, so with load
or smoke-test on the action loads and smoke-tests a single-platform build
first -- the runner's own platform if it is listed, the first one otherwise --
and builds (and pushes) every listed platform only once that passed; see
Build, test, push. load: "false" with
smoke-test: "false" builds them all in one go, untested.
The smoke test¶
With smoke-test: "true" the action runs its smoke-test.py inside the
freshly built image and checks that
- typesafe_carla's prebuilt CPython package loads without compiling
anything (
TYPESAFE_CARLA_PYCARLA_BUILD=0), and the officialcarlapackage is not installed, - the scenario package's
autoware_carla_scenario.scenariosentry point loads and registers at least one scenario, - Hydra discovers
AutowareScenarioSearchPathPlugin, so the package'sconf/directory reaches the config search path, - a config directory registered by a scenario package -- the framework's own
does not count -- exists and holds a YAML under its
scenario/group, so a wheel that lostconf/fails here rather than atscenario=<name>/..., and scenario --helpcomposes a config without a CARLA server.
Check 3 is worth its keep: hydra_plugins is a namespace package that an
editable install resolves via src/, so losing it from the wheel breaks every
scenario=<name>/… override in the image while leaving development untouched.
Build, test, push¶
The image is pushed only after its smoke test passed. The action builds it
with push: false (loaded into the local daemon), smoke-tests it, and then
runs the same build again with push: true: same builder, context, Dockerfile
and arguments, so every step is a cache hit and the pushed layers are the ones
the test ran. The digest output comes from that second run. For a
multi-platform platforms list the first build covers the one platform that
is smoke-tested and the second builds and pushes all of them, the tested one
from cache. With push: "true", load: "false" and smoke-test: "false"
there is nothing to wait for, and the action builds and pushes in one run.
Two caveats:
- The base images are referenced by tag. If an upstream tag is republished
between the two builds, the second one pulls the new base, misses the cache
from there on, and pushes an image that differs from the one tested. Pin
RUNTIME_BASEby digest (build-args: RUNTIME_BASE=python@sha256:…) to rule that out for the runtime base. The uv build-stage images (ghcr.io/astral-sh/uv:<UV_VERSION>…) take only a version, not a digest, so the same applies to them: a republished uv tag can change what the second build installs. - A multi-platform build takes longer -- the tested platform is built twice (once alone, then again from cache within the full build) and the others usually run emulated -- and only one platform is smoke-tested. For every platform tested, build each natively on its own runner as shown in Architectures.
Building by hand¶
ACTION=.github/actions/pack-scenario-image
# 1. Assemble a minimal build context (framework + scenario, nothing else).
"$ACTION/assemble-context.sh" \
--scenario ../my_scenario_package \
--out /tmp/scenario-ctx
# 2. Build. Nothing is compiled: the CARLA client's CPython package comes
# prebuilt in the typesafe-carla wheel.
docker build \
--file "$ACTION/Dockerfile" \
--tag my-scenario:latest \
/tmp/scenario-ctx
# 3. Verify.
docker run --rm -i --entrypoint python \
my-scenario:latest - < "$ACTION/smoke-test.py"
Running a scenario¶
The image holds the client, not the simulator, so point it at a CARLA server and give it the map files:
docker run --rm \
--network host \
--volume "$PWD/maps:/maps:ro" \
--volume "$PWD/outputs:/work/outputs" \
--env NISHISHINJUKU_XODR_PATH=/maps/nishishinjuku_carla.xodr \
--env NISHISHINJUKU_LANELET2_PATH=/maps/nishishinjuku.osm \
my-scenario:latest \
scenario=my_scenario/default map=nishishinjuku map.overwrite_xodr=false
map=nishishinjuku turns map.overwrite_xodr on: before loading the map it
copies map.xodr_path over the .xodr inside the server's CARLA installation,
the file NISHISHINJUKU_MAP_PATH names. The container cannot see that
installation, so the example turns the overwrite off and the server keeps the
road network its map already has; the mounted files are still what the scenario
reads. The map turns the overwrite on because NishishinjukuMap's assets ship
without a matching OpenDRIVE; to install it from a container on the server's
host, mount the server's file read-write and name it, in place of
map.overwrite_xodr=false ($SERVER_XODR is the path NISHISHINJUKU_MAP_PATH
would hold on the host):
--volume "$SERVER_XODR:/carla/NishishinjukuMap.xodr" \
--env NISHISHINJUKU_MAP_PATH=/carla/NishishinjukuMap.xodr \
The copy keeps the source's timestamps, which only the file's owner may set, so
the container's user (UID 1000, or --user) must own that file.
Without --network host, pass the server address explicitly with
server.host=<address> server.port=2000. Hydra writes under the working
directory, so /work needs a volume for the results to outlive the container —
the image runs as UID 1000, which must be able to write to it.