Sending a simulation to a remote node #
A simulation can run on another machine, a node, when its Compute request
names one: a workstation with a GPU, a server, the NAS under the desk. The run
record, FDTDRun or ElasticRun, is the one a run here takes, built and
checked here; the call is the same, fdtd_simulation(run, compute=...) or
elastic_fdtd_simulation(run, compute=...); and so is the record it returns.
Only those two functions, and the two video writers that take the same runs,
go to a node. An engine built directly, FDTD2D or ElasticFDTD2D, steps in
this process, where its fields are yours to read and write between steps, and
it refuses a request that names a host.
This page says what happens when a run goes to a node, how to name one, how to set up a Docker node or a Python node, what can go wrong and what each error says, how to follow a run, what each side trusts of the other, and when a run there gives the numbers a run here gives. The device itself, NumPy, CuPy or torch, is chosen as Where a simulation runs explains, and all of it holds on a node. Where heavy work runs maps the devices, the nodes and the encoders in one place.
1. What happens when a run goes to a node #
Everything that can refuse the call does so here first, before a connection
opens. The run checked every argument when it was built, by the code a run
here uses, so a wrong argument raises the TypeError or ValueError it
raises without a host. The function then refuses a run that cannot travel
(section 5 says which), and the hosts file must name the node.
Then one ssh process carries the whole run. There is no server to install
or keep running on the node: its ssh server is all it needs. The run travels
as data, field by field, never as code, and the node builds it again with the
library's constructors, so it checks itself again there. The video writers
travel too, their run, frame and video options as records: see
Writing animations and FDTD runs to video.
The node runs your own phonometry, as installed here, whatever is installed there. The client sends its files, those a wheel of it would hold, once per version: a node keeps each version under a digest of its files, which it computes again from what it receives, and a development checkout is a new version at each change. A node keeps the five packages used last and prunes the others, but never one used in the last ten minutes or one a run still holds, so a long run keeps its package however many others come and go.
The node runs the function on the device asked for, with no fallback of its
own, and sends back the probe histories and the snapshots, which are read here
into arrays sized from your request; a video writer sends its video's bytes
instead of snapshots. result.provenance.host names the node and
result.provenance.package_sha256 the package that ran; the rest of the
provenance is the node's: its NumPy, its array library, its GPU.
2. Naming a node: the hosts file #
A request names a node by a name of your choosing, and a hosts file says where that node is and how it runs a job. The file is read each time a run names a host, never at import:
| System | Where the hosts file is |
|---|---|
| Linux and macOS | ~/.config/phonometry/hosts.toml, or phonometry/hosts.toml under the directory XDG_CONFIG_HOME names, when that variable holds an absolute path |
| Windows | %APPDATA%\phonometry\hosts.toml |
| any system | the path PHONOMETRY_HOSTS_FILE names, when that variable is set; it wins over the rows above |
Each node is a table of its own. This file names a Docker node and a Python node:
# A GPU in the NAS, running the node image (section 3).
[hosts.nas]
ssh = "me@nas.local"
runner = "docker"
image = "phonometry-node:gpu"
label = "the GPU in the NAS"
timeout_s = 7200
# A workstation running a Python environment of its own (section 4).
[hosts.lab]
ssh = "lab" # a Host alias of ~/.ssh/config
runner = "python"
python = "venvs/node/bin/python" # relative to the home directory there
| Key | What it holds | Default |
|---|---|---|
ssh |
the destination ssh takes: user@address, or a Host alias of your ~/.ssh/config |
required |
runner |
how the node runs a job: "docker" or "python" |
required |
image |
for a Docker node, the image on the node | required there |
python |
for a Python node, its interpreter: an absolute path, or one relative to the home directory on the node | required there |
label |
how messages and the log name the node | its name |
timeout_s |
the longest a run there may take, in seconds | no limit |
A name holds letters, digits, ., _ and -, and starts with a letter or a
digit. A key the table does not list, an image for a Python node or a
python for a Docker one, or a python path that starts with ~ raises
ValueError naming the file: the client quotes the path in the command it
sends, so the node's shell would never expand the ~. A file saved with a
byte-order mark, as Windows Notepad saves UTF-8, reads like any other.
What ssh needs. The client runs the ssh of your machine, OpenSSH 7.6 or
newer, and a run never asks for anything: ssh runs in batch mode. It is the
ssh on PATH, run by the full path found there. On Windows, which looks for
a program in the working directory before PATH, an ssh there gives way to
the one on PATH, and one found only there is refused with a
ComputeRemoteError: put an OpenSSH client on PATH. Found in a working
directory that PATH itself lists, it runs as found. On Windows ssh also
starts with no console window, so a run from a notebook or from pythonw
opens none. Log in to the node with a key ssh can use without a prompt, from
ssh-agent or with no passphrase, and connect once by hand to accept the
node's host key. A user, a port, a key or a jump host belongs in a Host
alias of ~/.ssh/config. The options the client gives ssh on the command
line win over that file for what
a run cannot have: a RemoteCommand or a LocalCommand of the alias does
not run, nothing is forwarded (section 7), and no connection is shared:
every run is a connection of its own, whatever ControlMaster the alias
sets. Three settings stay the file's, because clients older than OpenSSH 8.7
refuse them on the command line, and an alias used as a node must leave them
at their defaults: SessionType none hangs a run until its timeout,
StdinNull yes fails it at once, and ForkAfterAuthentication yes detaches
the session out of the client's reach.
The request. host names the node, and timeout_s may bound the run; it
wins over the file's. Building the request reads no file and opens no
connection:
from phonometry import simulation
request = simulation.Compute(device="cupy", host="nas", timeout_s=3600)
print(request.host, request.timeout_s) # nas 3600.0
The run and the call are those of a run here:
run = simulation.FDTDRun(
c=343.0, dx=0.01, duration_s=0.05, shape=(400, 600),
sources=[simulation.GaussianPulse(ix=150, iy=200, half_width_s=3.0e-4)],
recording=simulation.FDTDRecording(probes=[(450, 200)]),
)
res = simulation.fdtd_simulation(run, compute=request)
print(res.provenance.host, res.provenance.gpu) # nas, and the card's name
elastic_fdtd_simulation(ElasticRun(...), compute=request) goes to the node
the same way. A whole script can name the node once, with a use() block or
with the PHONOMETRY_COMPUTE environment variable, whose grammar is
device[:index][@host][,option...]:
PHONOMETRY_COMPUTE=cupy@nas python study.py
PHONOMETRY_COMPUTE=cupy:1@nas,float32,fallback python study.py
A host is a name from the file, never user@address: the destination belongs
in the file, and Compute(host="me@nas") raises ValueError. An engine built
directly refuses a host wherever it comes from, a use() block and the
variable included, and its ValueError says what to pass instead: compute=
naming only the device, to step here, or the run to its simulation function,
to run the whole simulation there.
3. A Docker node #
A Docker node is a Linux machine whose login shell is a POSIX shell (sh,
bash, zsh: the command a client sends is quoted for one), with Docker
installed and your user in its docker group. A run on a node where your user
is not in that group fails, and its error gives the command that adds you. A
GPU needs the NVIDIA driver on the node as well, recent enough for the CUDA
runtime of the image, and the NVIDIA Container Toolkit.
Membership of the docker group is the same as root on the node: the
Docker daemon runs as root, and any member can start a container that reads
and writes every file of the machine, as Docker's own documentation warns.
The node's owner gives that membership only to an SSH account already
trusted with the whole machine.
The image is built on the node, from a checkout of the repository on your machine:
make node-image HOST=nas # a node with an NVIDIA GPU
make node-image HOST=nas TARGET=cpu # a node without one
HOST names an entry of your hosts file whose runner is docker. The command
sends the Dockerfile, docker/node/Dockerfile, and the few files it copies to
the node over ssh, docker build runs there, and the image takes the entry's
image as its tag. Nothing is published to a registry, and a rebuild with
nothing changed comes from Docker's cache.
Both builds hold Python 3.13 with the library's pinned dependencies and the libraries its figures need, the DejaVu fonts the figures are drawn with, and ffmpeg, a static build the Dockerfile pins by URL and SHA-256, with SVT-AV1, NVENC, VP9 and dav1d, so a node needs no ffmpeg of its own. The GPU build adds the CUDA runtime and CuPy 14.2.0, the release the GPU proof of Where a simulation runs ran on. Each base image is pinned by digest.
Each run gets a container of its own, removed when the run ends, with no
network, running as your user on the node, so whatever it writes is yours. Its
packages live in ~/.cache/phonometry/remote on the node (under
XDG_CACHE_HOME when that variable holds an absolute path), which the
container mounts. A run takes one GPU, the one device_index picks. Docker
never pulls the image: a node without it fails the run, and the error says to
build it.
The hosts file may name an image of your own. The client runs the bootstrap a
node starts with as the container's entrypoint, python on the image's PATH
running /usr/local/share/phonometry-node/bootstrap.py, so such an image needs
a python on its PATH and that file at that path (the Dockerfile copies it
from the package), with Python, NumPy and SciPy at the floors of section 4,
and CuPy or torch for a GPU. Its own ENTRYPOINT does not run: nothing it
would print can break the conversation with the client, and an environment it
would activate is not active.
4. A Python node #
A Python node runs a job in a Python environment of its own, with no Docker.
It is a Linux or macOS machine whose login shell is a POSIX shell, as for
Docker. Windows works as the client, not as a node: its OpenSSH server runs
the command through cmd.exe, which reads the quoting another way.
The environment needs Python 3.12 or later, with NumPy 2.0.2 or later and
SciPy 1.14.0 or later, the library's floors, and CuPy or torch for a GPU,
installed as Where a simulation runs says.
phonometry need not be installed there, and an installed one is not used: the
interpreter runs isolated (-I), on the package the client sent, and the node
checks that the phonometry it imported is that one. An environment below a
floor refuses the run, and the error names the floor and says to upgrade the
environment.
python3 -m venv ~/venvs/node
~/venvs/node/bin/pip install "numpy>=2.0.2" "scipy>=1.14.0"
Quote each requirement: unquoted, the >= is a shell redirection. The hosts
file then names the interpreter by its path, venvs/node/bin/python from the
home directory or the absolute path. The client runs that python directly,
with no activation script first, so the environment must work without one.
A node that writes videos
(Writing animations and FDTD runs to video) needs two things more:
matplotlib in that environment, at the version the library's plot extra
asks for, and ffmpeg on the PATH the command ssh runs there sees. The
Docker image of section 3 holds both.
5. When something goes wrong #
| What went wrong | What is raised | When |
|---|---|---|
| an argument of the run | TypeError or ValueError |
when the run is built, as for a run here |
a run that cannot travel: a source whose waveform is a callable other than the .value of a GaussianPulse, CWSource or SignalSource, or a whole number outside the range of an int64 |
TypeError, naming the source or the field |
when the function is called, before the hosts file is read |
| a host the file does not name, or no hosts file | KeyError |
before ssh starts |
| a hosts file that cannot be read or is not TOML, or a malformed entry | ValueError, naming the file |
before ssh starts |
no ssh client on PATH, even on Windows with one in the working directory, Pyodide, a package that cannot be read from disk or holds a file no node would write, or a job too long for the header a node reads (a run of several thousand point sources) |
ComputeRemoteError |
before ssh starts |
| anything that fails in the connection or on the node | ComputeRemoteError |
when it fails |
The message of a ComputeRemoteError says what happened and, when it can
tell, what to do: install an OpenSSH client here, build the image on the node
(make node-image HOST=nas),
join the docker group there, install the NVIDIA Container Toolkit, lift the
environment to the floors or rebuild the image, add your key to the node's
~/.ssh/authorized_keys, accept its host key, make a login script print
nothing when the shell is not interactive, or rename a file of a development
checkout whose name no node would write. Its host holds the node's name, and
its node_traceback the traceback the node printed when the run raised
there, which is also attached as a note, so a traceback printed here shows
it. A connection that broke adds the last lines the node wrote to its
standard error as a note too.
With fallback=True a ComputeRemoteError becomes a ComputeFallbackWarning,
whose message carries the error, and the run steps on NumPy here, in the dtype
asked for; its provenance says so, with no host and the device "cpu". Every
ComputeRemoteError falls back, those raised here before ssh starts
included; a wrong argument, an unknown host or a malformed hosts file raises
with or without it.
Warnings the run raises on the node are raised here, at the line of your code
that called the function, with their own class: phonometry's as published,
Python's and NumPy's as themselves. A warning of a library this process has
not imported, or of a class that cannot be built from its message, comes back
as a RuntimeWarning whose message starts with the class's path.
An interrupt, Ctrl-C or the stop button of a notebook, stops ssh at once and
reaches your code as KeyboardInterrupt, fallback=True or not. The node
sees its input end, and its run ends at most 30 seconds after its client
closes the connection, usually at its next step. A client that vanishes
without closing it (a laptop put to sleep, a network that drops) is noticed
only when the node's ssh server notices. Set its keep-alive in the node's
sshd_config, to the values the client's own keep-alive uses, and reload the
server:
ClientAliveInterval 15
ClientAliveCountMax 4
With them the server notices a vanished client in about a minute. Without them the run goes on to its end, and a GPU stays busy for it.
6. Progress and timeouts #
A remote run reports to the phonometry.simulation logger, which says nothing
until you configure logging:
import logging
logging.basicConfig(level=logging.INFO)
At INFO the logger says when a run starts, when the package is sent (its
digest and its size), the step the node has reached as the run goes, and when
the run is done, each line led by the node's label; at DEBUG it adds what
the node writes to its standard error.
timeout_s, from the request or else from the hosts file, bounds a whole run,
from connecting to the result, measured here: past it ssh is stopped and the
run raises ComputeRemoteError, or falls back with fallback=True. Without
one a run may take as long as it takes. Two bounds hold whatever it says: ssh
notices a dead connection in about a minute, and gives up connecting after
30 seconds.
7. Security, in both directions #
SSH is the boundary. Whoever can log in to the node can run code there, and a
remote run does exactly that with the package you send, so give the node the
trust you give any machine you log in to. ssh is started with no terminal,
with no agent, X11 or port forwarding, and without delegating your Kerberos
credentials, whatever ~/.ssh/config says for the host. Besides the package
and the job, it may send the environment variables your ssh configuration
names with SendEnv or SetEnv (on Debian and Ubuntu, for example LANG and
the LC_* variables), as far as the node's AcceptEnv allows.
A job carries no code: the node runs the package you sent, on its own Python and libraries. The job names one of the two simulation functions or of the two video writers and carries its run as data, which the node builds again with the library's constructors, and a package is checked against its digest before it is stored. A Docker node's container has no network.
The client does not trust the bytes the node sends. Every header and every array is checked against what the request implies, its name, its dtype, its shape and its length, before it is read; nothing is unpickled; no module the node names is imported unless it is phonometry's own; and the node's words (its errors, tracebacks, log lines and warnings) reach your terminal with their control characters written as escapes, so a node cannot send an escape sequence to your terminal.
8. When a run there equals a run here #
A run on a node steps the same code as a run here, on the node's own NumPy and its own CPU or GPU. On a CPU it gives the same numbers bit for bit only with the same pins on the same CPU: the same Python, NumPy and SciPy, on the same kind of processor and C library. A CI job proves it on every pull request, against a node on the same machine. Between two machines, NumPy's vectorised functions, such as the exponential of the sponge profile, may round differently by one unit in the last place, each CPU taking the instructions it has. On a GPU, Where a simulation runs states what the proof on the GPU shows.
9. For library developers #
The remote route's tests run on every pull request through a stand-in for
ssh, on Linux, macOS and Windows, and a CI job runs a real node on localhost:
it builds the node image for a CPU, serves it through an ssh server and holds
a run there bit for bit to the run here. make node-image HOST=<name> builds
the image on your node from your checkout, with the Dockerfile and the
bootstrap as you changed them. After a change to an engine, make gpu-parity,
run from your checkout in that image on the node's GPU, renews the GPU proof
in gpu-parity.lock.
Rendering the documentation clips on a node #
The documentation's animated clips, most of them simulations stepped on the
engines of this page, can be rendered on the same node: with the node in the
hosts file, PHONOMETRY_COMPUTE=cupy@<name> make animations sends the
repository's committed code there, renders the clips two at a time with the
GPU and NVENC, and brings them back checked. The
contributing guide
explains it, and how to render them here on a CPU, which gives clips fit to
publish too.
What this guide covers #
Covered. Running fdtd_simulation and elastic_fdtd_simulation on
another machine over SSH, on a Docker node built from the repository's image
or on a Python environment, with or without a GPU; the hosts file that names
a node, and the host of Compute and of PHONOMETRY_COMPUTE; what the
client and the node check, each of what the other sends; the errors a remote
run raises, the fallback to NumPy here, the warnings the node raises, progress
on the phonometry.simulation logger, timeouts and interrupts; and when a
remote run gives the numbers of a run here.
Not covered. An engine built directly, FDTD2D or ElasticFDTD2D,
steps in this process and does not go to a node, and a node runs the two
simulation functions and the two video writers only, never code of the
caller's choosing. ssh is the only transport. A node has no scheduler or queue
of its own: two runs sent at the same time run side by side. The node image is
built on the node and published to no registry, and a Windows machine cannot
be a node. A node encodes the video of a run it steps, but no animation drawn
here: io.write_video runs on this machine only.
See also #
- Where a simulation runs: NumPy, CuPy and torch: the devices a run steps on, here or on a node, and what is promised about the numbers on each.
- 2D FDTD wave simulation: the acoustic run, its sources, probes and boundaries.
- Elastic waves and fluid-solid coupling: the elastic run, which goes to a node the same way.
- API reference:
simulation.compute,simulation.fdtd,simulation.elastic_fdtd.
Quick answers #
How do I run a simulation on another machine? #
Name the machine in a hosts file, give it the node image
(make node-image HOST=nas) or a Python environment with NumPy and SciPy, and
pass compute=simulation.Compute(device="cupy", host="nas") to
fdtd_simulation or elastic_fdtd_simulation, or set
PHONOMETRY_COMPUTE=cupy@nas. The run, the call and the result are those of a
run here, and result.provenance.host names the node.
Does a remote run give the same numbers? #
With the same pins on the same CPU, bit for bit, as a CI job proves on every pull request. Between two machines, NumPy may round a vectorised function, such as the exponential of the sponge profile, differently by one unit in the last place. On a GPU, the proof on the compute page states how the fields compare with NumPy's.
What does the node need? #
An ssh server you can log in to with a key, on Linux or macOS with a POSIX
login shell, and either Docker with the node image (and, for a GPU, the
NVIDIA driver and the NVIDIA Container Toolkit) or a Python 3.12 or later with
NumPy and SciPy at the library's floors (and CuPy or torch for a GPU). To
write videos, a Python node also needs matplotlib, and ffmpeg on its
PATH; the Docker image holds both. phonometry itself need not be installed there:
the client sends its own.