[READ-ONLY] Mirror of https://github.com/jmrplens/phonometry. Standards-conformant acoustics library for Python: measurement and metrology, docs.jmrp.io/phonometry
phonometry docs simulation remote-node.md
23 kB · 444 lines
Markdown
at main

← Documentation index

Sending a simulation to a remote node #

A simulation can run on another machine, a node, when its Compute request names one: a workstation with a GPU, a server, the NAS under the desk. The run record, FDTDRun or ElasticRun, is the one a run here takes, built and checked here; the call is the same, fdtd_simulation(run, compute=...) or elastic_fdtd_simulation(run, compute=...); and so is the record it returns. Only those two functions, and the two video writers that take the same runs, go to a node. An engine built directly, FDTD2D or ElasticFDTD2D, steps in this process, where its fields are yours to read and write between steps, and it refuses a request that names a host.

This page says what happens when a run goes to a node, how to name one, how to set up a Docker node or a Python node, what can go wrong and what each error says, how to follow a run, what each side trusts of the other, and when a run there gives the numbers a run here gives. The device itself, NumPy, CuPy or torch, is chosen as Where a simulation runs explains, and all of it holds on a node. Where heavy work runs maps the devices, the nodes and the encoders in one place.

1. What happens when a run goes to a node #

Everything that can refuse the call does so here first, before a connection opens. The run checked every argument when it was built, by the code a run here uses, so a wrong argument raises the TypeError or ValueError it raises without a host. The function then refuses a run that cannot travel (section 5 says which), and the hosts file must name the node.

Then one ssh process carries the whole run. There is no server to install or keep running on the node: its ssh server is all it needs. The run travels as data, field by field, never as code, and the node builds it again with the library's constructors, so it checks itself again there. The video writers travel too, their run, frame and video options as records: see Writing animations and FDTD runs to video.

The node runs your own phonometry, as installed here, whatever is installed there. The client sends its files, those a wheel of it would hold, once per version: a node keeps each version under a digest of its files, which it computes again from what it receives, and a development checkout is a new version at each change. A node keeps the five packages used last and prunes the others, but never one used in the last ten minutes or one a run still holds, so a long run keeps its package however many others come and go.

The node runs the function on the device asked for, with no fallback of its own, and sends back the probe histories and the snapshots, which are read here into arrays sized from your request; a video writer sends its video's bytes instead of snapshots. result.provenance.host names the node and result.provenance.package_sha256 the package that ran; the rest of the provenance is the node's: its NumPy, its array library, its GPU.

2. Naming a node: the hosts file #

A request names a node by a name of your choosing, and a hosts file says where that node is and how it runs a job. The file is read each time a run names a host, never at import:

System Where the hosts file is
Linux and macOS ~/.config/phonometry/hosts.toml, or phonometry/hosts.toml under the directory XDG_CONFIG_HOME names, when that variable holds an absolute path
Windows %APPDATA%\phonometry\hosts.toml
any system the path PHONOMETRY_HOSTS_FILE names, when that variable is set; it wins over the rows above

Each node is a table of its own. This file names a Docker node and a Python node:

# A GPU in the NAS, running the node image (section 3).
[hosts.nas]
ssh = "me@nas.local"
runner = "docker"
image = "phonometry-node:gpu"
label = "the GPU in the NAS"
timeout_s = 7200

# A workstation running a Python environment of its own (section 4).
[hosts.lab]
ssh = "lab"                         # a Host alias of ~/.ssh/config
runner = "python"
python = "venvs/node/bin/python"    # relative to the home directory there
Key What it holds Default
ssh the destination ssh takes: user@address, or a Host alias of your ~/.ssh/config required
runner how the node runs a job: "docker" or "python" required
image for a Docker node, the image on the node required there
python for a Python node, its interpreter: an absolute path, or one relative to the home directory on the node required there
label how messages and the log name the node its name
timeout_s the longest a run there may take, in seconds no limit

A name holds letters, digits, ., _ and -, and starts with a letter or a digit. A key the table does not list, an image for a Python node or a python for a Docker one, or a python path that starts with ~ raises ValueError naming the file: the client quotes the path in the command it sends, so the node's shell would never expand the ~. A file saved with a byte-order mark, as Windows Notepad saves UTF-8, reads like any other.

What ssh needs. The client runs the ssh of your machine, OpenSSH 7.6 or newer, and a run never asks for anything: ssh runs in batch mode. It is the ssh on PATH, run by the full path found there. On Windows, which looks for a program in the working directory before PATH, an ssh there gives way to the one on PATH, and one found only there is refused with a ComputeRemoteError: put an OpenSSH client on PATH. Found in a working directory that PATH itself lists, it runs as found. On Windows ssh also starts with no console window, so a run from a notebook or from pythonw opens none. Log in to the node with a key ssh can use without a prompt, from ssh-agent or with no passphrase, and connect once by hand to accept the node's host key. A user, a port, a key or a jump host belongs in a Host alias of ~/.ssh/config. The options the client gives ssh on the command line win over that file for what a run cannot have: a RemoteCommand or a LocalCommand of the alias does not run, nothing is forwarded (section 7), and no connection is shared: every run is a connection of its own, whatever ControlMaster the alias sets. Three settings stay the file's, because clients older than OpenSSH 8.7 refuse them on the command line, and an alias used as a node must leave them at their defaults: SessionType none hangs a run until its timeout, StdinNull yes fails it at once, and ForkAfterAuthentication yes detaches the session out of the client's reach.

The request. host names the node, and timeout_s may bound the run; it wins over the file's. Building the request reads no file and opens no connection:

from phonometry import simulation

request = simulation.Compute(device="cupy", host="nas", timeout_s=3600)
print(request.host, request.timeout_s)   # nas 3600.0

The run and the call are those of a run here:

run = simulation.FDTDRun(
    c=343.0, dx=0.01, duration_s=0.05, shape=(400, 600),
    sources=[simulation.GaussianPulse(ix=150, iy=200, half_width_s=3.0e-4)],
    recording=simulation.FDTDRecording(probes=[(450, 200)]),
)
res = simulation.fdtd_simulation(run, compute=request)
print(res.provenance.host, res.provenance.gpu)   # nas, and the card's name

elastic_fdtd_simulation(ElasticRun(...), compute=request) goes to the node the same way. A whole script can name the node once, with a use() block or with the PHONOMETRY_COMPUTE environment variable, whose grammar is device[:index][@host][,option...]:

PHONOMETRY_COMPUTE=cupy@nas python study.py
PHONOMETRY_COMPUTE=cupy:1@nas,float32,fallback python study.py

A host is a name from the file, never user@address: the destination belongs in the file, and Compute(host="me@nas") raises ValueError. An engine built directly refuses a host wherever it comes from, a use() block and the variable included, and its ValueError says what to pass instead: compute= naming only the device, to step here, or the run to its simulation function, to run the whole simulation there.

3. A Docker node #

A Docker node is a Linux machine whose login shell is a POSIX shell (sh, bash, zsh: the command a client sends is quoted for one), with Docker installed and your user in its docker group. A run on a node where your user is not in that group fails, and its error gives the command that adds you. A GPU needs the NVIDIA driver on the node as well, recent enough for the CUDA runtime of the image, and the NVIDIA Container Toolkit.

Membership of the docker group is the same as root on the node: the Docker daemon runs as root, and any member can start a container that reads and writes every file of the machine, as Docker's own documentation warns. The node's owner gives that membership only to an SSH account already trusted with the whole machine.

The image is built on the node, from a checkout of the repository on your machine:

make node-image HOST=nas               # a node with an NVIDIA GPU
make node-image HOST=nas TARGET=cpu    # a node without one

HOST names an entry of your hosts file whose runner is docker. The command sends the Dockerfile, docker/node/Dockerfile, and the few files it copies to the node over ssh, docker build runs there, and the image takes the entry's image as its tag. Nothing is published to a registry, and a rebuild with nothing changed comes from Docker's cache.

Both builds hold Python 3.13 with the library's pinned dependencies and the libraries its figures need, the DejaVu fonts the figures are drawn with, and ffmpeg, a static build the Dockerfile pins by URL and SHA-256, with SVT-AV1, NVENC, VP9 and dav1d, so a node needs no ffmpeg of its own. The GPU build adds the CUDA runtime and CuPy 14.2.0, the release the GPU proof of Where a simulation runs ran on. Each base image is pinned by digest.

Each run gets a container of its own, removed when the run ends, with no network, running as your user on the node, so whatever it writes is yours. Its packages live in ~/.cache/phonometry/remote on the node (under XDG_CACHE_HOME when that variable holds an absolute path), which the container mounts. A run takes one GPU, the one device_index picks. Docker never pulls the image: a node without it fails the run, and the error says to build it.

The hosts file may name an image of your own. The client runs the bootstrap a node starts with as the container's entrypoint, python on the image's PATH running /usr/local/share/phonometry-node/bootstrap.py, so such an image needs a python on its PATH and that file at that path (the Dockerfile copies it from the package), with Python, NumPy and SciPy at the floors of section 4, and CuPy or torch for a GPU. Its own ENTRYPOINT does not run: nothing it would print can break the conversation with the client, and an environment it would activate is not active.

4. A Python node #

A Python node runs a job in a Python environment of its own, with no Docker. It is a Linux or macOS machine whose login shell is a POSIX shell, as for Docker. Windows works as the client, not as a node: its OpenSSH server runs the command through cmd.exe, which reads the quoting another way.

The environment needs Python 3.12 or later, with NumPy 2.0.2 or later and SciPy 1.14.0 or later, the library's floors, and CuPy or torch for a GPU, installed as Where a simulation runs says. phonometry need not be installed there, and an installed one is not used: the interpreter runs isolated (-I), on the package the client sent, and the node checks that the phonometry it imported is that one. An environment below a floor refuses the run, and the error names the floor and says to upgrade the environment.

python3 -m venv ~/venvs/node
~/venvs/node/bin/pip install "numpy>=2.0.2" "scipy>=1.14.0"

Quote each requirement: unquoted, the >= is a shell redirection. The hosts file then names the interpreter by its path, venvs/node/bin/python from the home directory or the absolute path. The client runs that python directly, with no activation script first, so the environment must work without one.

A node that writes videos (Writing animations and FDTD runs to video) needs two things more: matplotlib in that environment, at the version the library's plot extra asks for, and ffmpeg on the PATH the command ssh runs there sees. The Docker image of section 3 holds both.

5. When something goes wrong #

What went wrong What is raised When
an argument of the run TypeError or ValueError when the run is built, as for a run here
a run that cannot travel: a source whose waveform is a callable other than the .value of a GaussianPulse, CWSource or SignalSource, or a whole number outside the range of an int64 TypeError, naming the source or the field when the function is called, before the hosts file is read
a host the file does not name, or no hosts file KeyError before ssh starts
a hosts file that cannot be read or is not TOML, or a malformed entry ValueError, naming the file before ssh starts
no ssh client on PATH, even on Windows with one in the working directory, Pyodide, a package that cannot be read from disk or holds a file no node would write, or a job too long for the header a node reads (a run of several thousand point sources) ComputeRemoteError before ssh starts
anything that fails in the connection or on the node ComputeRemoteError when it fails

The message of a ComputeRemoteError says what happened and, when it can tell, what to do: install an OpenSSH client here, build the image on the node (make node-image HOST=nas), join the docker group there, install the NVIDIA Container Toolkit, lift the environment to the floors or rebuild the image, add your key to the node's ~/.ssh/authorized_keys, accept its host key, make a login script print nothing when the shell is not interactive, or rename a file of a development checkout whose name no node would write. Its host holds the node's name, and its node_traceback the traceback the node printed when the run raised there, which is also attached as a note, so a traceback printed here shows it. A connection that broke adds the last lines the node wrote to its standard error as a note too.

With fallback=True a ComputeRemoteError becomes a ComputeFallbackWarning, whose message carries the error, and the run steps on NumPy here, in the dtype asked for; its provenance says so, with no host and the device "cpu". Every ComputeRemoteError falls back, those raised here before ssh starts included; a wrong argument, an unknown host or a malformed hosts file raises with or without it.

Warnings the run raises on the node are raised here, at the line of your code that called the function, with their own class: phonometry's as published, Python's and NumPy's as themselves. A warning of a library this process has not imported, or of a class that cannot be built from its message, comes back as a RuntimeWarning whose message starts with the class's path.

An interrupt, Ctrl-C or the stop button of a notebook, stops ssh at once and reaches your code as KeyboardInterrupt, fallback=True or not. The node sees its input end, and its run ends at most 30 seconds after its client closes the connection, usually at its next step. A client that vanishes without closing it (a laptop put to sleep, a network that drops) is noticed only when the node's ssh server notices. Set its keep-alive in the node's sshd_config, to the values the client's own keep-alive uses, and reload the server:

ClientAliveInterval 15
ClientAliveCountMax 4

With them the server notices a vanished client in about a minute. Without them the run goes on to its end, and a GPU stays busy for it.

6. Progress and timeouts #

A remote run reports to the phonometry.simulation logger, which says nothing until you configure logging:

import logging

logging.basicConfig(level=logging.INFO)

At INFO the logger says when a run starts, when the package is sent (its digest and its size), the step the node has reached as the run goes, and when the run is done, each line led by the node's label; at DEBUG it adds what the node writes to its standard error.

timeout_s, from the request or else from the hosts file, bounds a whole run, from connecting to the result, measured here: past it ssh is stopped and the run raises ComputeRemoteError, or falls back with fallback=True. Without one a run may take as long as it takes. Two bounds hold whatever it says: ssh notices a dead connection in about a minute, and gives up connecting after 30 seconds.

7. Security, in both directions #

SSH is the boundary. Whoever can log in to the node can run code there, and a remote run does exactly that with the package you send, so give the node the trust you give any machine you log in to. ssh is started with no terminal, with no agent, X11 or port forwarding, and without delegating your Kerberos credentials, whatever ~/.ssh/config says for the host. Besides the package and the job, it may send the environment variables your ssh configuration names with SendEnv or SetEnv (on Debian and Ubuntu, for example LANG and the LC_* variables), as far as the node's AcceptEnv allows.

A job carries no code: the node runs the package you sent, on its own Python and libraries. The job names one of the two simulation functions or of the two video writers and carries its run as data, which the node builds again with the library's constructors, and a package is checked against its digest before it is stored. A Docker node's container has no network.

The client does not trust the bytes the node sends. Every header and every array is checked against what the request implies, its name, its dtype, its shape and its length, before it is read; nothing is unpickled; no module the node names is imported unless it is phonometry's own; and the node's words (its errors, tracebacks, log lines and warnings) reach your terminal with their control characters written as escapes, so a node cannot send an escape sequence to your terminal.

8. When a run there equals a run here #

A run on a node steps the same code as a run here, on the node's own NumPy and its own CPU or GPU. On a CPU it gives the same numbers bit for bit only with the same pins on the same CPU: the same Python, NumPy and SciPy, on the same kind of processor and C library. A CI job proves it on every pull request, against a node on the same machine. Between two machines, NumPy's vectorised functions, such as the exponential of the sponge profile, may round differently by one unit in the last place, each CPU taking the instructions it has. On a GPU, Where a simulation runs states what the proof on the GPU shows.

9. For library developers #

The remote route's tests run on every pull request through a stand-in for ssh, on Linux, macOS and Windows, and a CI job runs a real node on localhost: it builds the node image for a CPU, serves it through an ssh server and holds a run there bit for bit to the run here. make node-image HOST=<name> builds the image on your node from your checkout, with the Dockerfile and the bootstrap as you changed them. After a change to an engine, make gpu-parity, run from your checkout in that image on the node's GPU, renews the GPU proof in gpu-parity.lock.

Rendering the documentation clips on a node #

The documentation's animated clips, most of them simulations stepped on the engines of this page, can be rendered on the same node: with the node in the hosts file, PHONOMETRY_COMPUTE=cupy@<name> make animations sends the repository's committed code there, renders the clips two at a time with the GPU and NVENC, and brings them back checked. The contributing guide explains it, and how to render them here on a CPU, which gives clips fit to publish too.

What this guide covers #

Covered. Running fdtd_simulation and elastic_fdtd_simulation on another machine over SSH, on a Docker node built from the repository's image or on a Python environment, with or without a GPU; the hosts file that names a node, and the host of Compute and of PHONOMETRY_COMPUTE; what the client and the node check, each of what the other sends; the errors a remote run raises, the fallback to NumPy here, the warnings the node raises, progress on the phonometry.simulation logger, timeouts and interrupts; and when a remote run gives the numbers of a run here.

Not covered. An engine built directly, FDTD2D or ElasticFDTD2D, steps in this process and does not go to a node, and a node runs the two simulation functions and the two video writers only, never code of the caller's choosing. ssh is the only transport. A node has no scheduler or queue of its own: two runs sent at the same time run side by side. The node image is built on the node and published to no registry, and a Windows machine cannot be a node. A node encodes the video of a run it steps, but no animation drawn here: io.write_video runs on this machine only.

See also #

Quick answers #

How do I run a simulation on another machine? #

Name the machine in a hosts file, give it the node image (make node-image HOST=nas) or a Python environment with NumPy and SciPy, and pass compute=simulation.Compute(device="cupy", host="nas") to fdtd_simulation or elastic_fdtd_simulation, or set PHONOMETRY_COMPUTE=cupy@nas. The run, the call and the result are those of a run here, and result.provenance.host names the node.

Does a remote run give the same numbers? #

With the same pins on the same CPU, bit for bit, as a CI job proves on every pull request. Between two machines, NumPy may round a vectorised function, such as the exponential of the sponge profile, differently by one unit in the last place. On a GPU, the proof on the compute page states how the fields compare with NumPy's.

What does the node need? #

An ssh server you can log in to with a key, on Linux or macOS with a POSIX login shell, and either Docker with the node image (and, for a GPU, the NVIDIA driver and the NVIDIA Container Toolkit) or a Python 3.12 or later with NumPy and SciPy at the library's floors (and CuPy or torch for a GPU). To write videos, a Python node also needs matplotlib, and ffmpeg on its PATH; the Docker image holds both. phonometry itself need not be installed there: the client sends its own.