The program interface¶
How qccompute runs programs: the protocol a program implements, the context a runner gives each calculation, sessions, failures, the runner, the store, and worker processes. This is the design as built; the API pages document each name.
The layers¶
workflows, apps, analysis read and write qcdata; call qccompute
─────────────────────────────────────────────────────────────────────────
qcdata data only: inputs, outputs, Frame, Request, Accuracy, ProgramError,
conventions, the calculation key, the save format
qccompute the Program protocol (qccompute.program) and the Runner: discovery,
planning, placement, resources, lists, sessions, workers, the store hook
─────────────────────────────────────────────────────────────────────────
programs implement the protocol; a program without subprograms imports only qcdata
qccodec external programs (xtb, ORCA, Psi4, CREST, ...) as programs: input files,
subprocesses, output capture, file collection, parsers
qccompute depends on qcdata only. qccodec is the optional add-on for external programs
(pip install qccompute[qccodec]) and does not import qccompute to run them.
Programs¶
There is one kind of program. A program maps one ProgramInput to one ProgramOutput
and may have subprograms (ProgramSpecs in its input) that it runs through the runner.
Running many programs, with logic between them, is a workflow: plain Python.
A program registers an instance under the qccompute.programs entry point group, or in a
process with qccompute.register_program (a registered program that runs in a worker
reaches it by its module attribute, else pickled; see Workers):
[project.entry-points."qccompute.programs"]
myprogram = "mypackage.qc:program"
The object implements qccompute.program.Program, structurally:
| member | what it is |
|---|---|
name |
the program's name, as in ProgramInput.program |
version() |
the program's version |
capabilities() |
calctypes, methods with their calctypes, sessions, batches, in_process, thread_safe; a Capabilities or a plain mapping |
settings_schema(calctype) |
the JSON Schema of keywords and of resolved.settings |
record_schema(calctype) |
the JSON Schema of ProgramOutput.record |
plan(inp) |
a dry run: validates (kind input with a did-you-mean for unknown keys), resolves every "auto", and says what the calculation needs (Resources: GPUs; memory, informational) |
compute(inp, ctx=None) |
runs the calculation; returns the successful output or raises ProgramError |
numerics(spec) (optional) |
a string that changes whenever results for an unchanged input may change; the version by default |
open_session(inp, ctx=None, calctypes=None) (optional) |
a session, usually the program's own session class |
compute_batch(inputs, ctx=None) (optional) |
many inputs at once; one output per input, in order, failures in place |
A qcdata.FileInput that is not a ProgramInput (a command with its files, whose
program is the command) runs under the program registered as files, the one name the
protocol reserves; qccodec registers it, and its numerics identity is the run's identity
({"files": ...}).
What a program returns for capabilities and plan may be a plain mapping, which
qccompute validates (capabilities_of, plan_of): it is the same document a program's
command line prints and its worker sends. numerics takes the spec it would run (the
input, or a subprogram's ProgramSpec), so a program of several methods can give each
its own identity: a numerics epoch, an executable's sha256, a model's checksum.
qccompute.program.FunctionProgram(name, energy_gradient, hessian=None, *, version=None,
takes_input=False, settings=None) is the smallest complete program: functions of the
positions, in process and thread-safe, taking the shared threads and accuracy. With
takes_input=True the functions also get the input (structure, model, keywords), and
settings= declares the JSON Schema of the keywords they read, so one program stands
in for an engine on any molecule. Its numerics identity is a digest of what the
functions compute with (code, constants, defaults, closures, the globals they read),
unless version names it.
The context¶
compute's second argument is a RunContext, which only a runner passes, and a runner
always passes one. It carries the calculation's thread count, GPUs and memory and the way
to run subprograms:
threads: the calculation's thread count: itsthreadskeyword, else its caller's whole count (1 when it is given GPUs), as Threads explains. The program uses at most that many; a larger request of its own is aresource_cappedwarning.gpus: as many whole GPUs as its plan asks for (plan.resources.gpus), of the runner's or the parent's: a program whose subprograms use GPUs plans theirs and passes them on. While the program runs, the runner holds the process's BLAS and OpenMP pools at its threads, or at one thread when its input names subprograms (below, Thread pools).device_ordinals(): the granted GPUs as this process's CUDA device ordinals, so every program maps a grant alike.gpusnames them asCUDA_VISIBLE_DEVICESdoes (indices orGPU-/MIG-UUIDs). When the process'sCUDA_VISIBLE_DEVICESis set, a GPU's ordinal is its position there: a worker is started withCUDA_VISIBLE_DEVICESset to its grant, so its GPUs are 0, 1, ...; in the runner's process GPU"2"ofCUDA_VISIBLE_DEVICES=1,2is ordinal 1. When it is unset, an index is its own ordinal and a UUID is the indexnvidia-smigives it.nvidia-sminumbers GPUs in PCI bus order and CUDA by default may not, so a runner that finds its GPUs withnvidia-smisetsCUDA_DEVICE_ORDER=PCI_BUS_ID(unless it is set) when it is made, for its process and its workers; a process that starts CUDA before it makes a runner sets it itself. A dispatcher'sCUDA_VISIBLE_DEVICESis never changed. A granted GPU the process cannot see fails with kindenvironment.memory_gib,device_memory_gib: host and GPU memory (GiB), as the runner grants it (below, Memory).timeout: the seconds left. A program that starts processes kills them when it runs out and fails with kindtimeout; one that loops checks it. The runner kills a program in a worker process that runs out, but cannot stop one in its own process (Python cannot stop a thread): there the limit is the program's to honour, and the runner starts none of its subprograms once it has run out (ctx.compute,ctx.sessionand its child sessions' requests fail with kindtimeout), so a driver stops at its next step. For a hard limit on any program,isolation="process".plan: the runner's plan of the input, so the program need not plan again.progress: where the program reports what it is doing while it runs, when the caller listens (below, Progress); None when nobody does, so a program checks it before building an event.stop_requested(): whether the caller asked the calculation to stop (below, Stopping). A program that works in steps checks it before each step.scratch: a directory, made on first use and removed when the call returns. In a worker it is made in a directory the worker's client gives it (--scratch) and removes when it closes the worker, so a calculation whose worker is killed (a timeout, a cancel) leaves no scratch behind.QCCOMPUTE_KEEP_SCRATCH=1keeps every one, for debugging; a killed worker's failure then names where its scratch is.compute(inputs, raise_exc=True)andsession(spec, structure, threads=, calctypes=): subprograms, under the calculation's thread count. They run only programs the calculation's input names, its own and its subprograms' recursively, and every program a subprogram's input names must be among them; another is refused (kindinput, codeunnamed_subprogram), so a run's identities cover every program it used. The refusal fails the calculation, whatever the program returns or raises afterwards: a store then reuses it, as any input error, rather than rerunning the work. A program that decides at run time which programs to run (a path method that minimizes its ends) names each in its input's subprograms.
ctx is None means a direct call: the program uses its own settings. A program with
subprograms has no route of its own to programs, so called directly it returns
qccompute.compute(inp), and the default runner runs it with a context. The runner finds
programs by name: one that is neither installed nor registered fails with kind
environment, and the message says how to register it.
The runner never changes a number by itself. A grant may change a program's resolved
choices (a storage scheme picked from the memory granted, a precision from the device);
the program records each one in resolved.
Progress¶
submit, compute and session take progress=, a callable that
receives small JSON events while calculations run. Programs choose what to emit (qcopt:
one event per step); the runner only delivers them, and nothing is stored.
- A program calls
ctx.progress(event)with a JSON object of its own: plain values, qcdata's units, small enough to send every step (a step number, an energy, the largest gradient component; positions where a live view needs them). A shared key:event, a snake_case name of what happened (step). - The caller receives the event with
pathadded: the program names from the caller's input down to the program that emitted it (["neb", "qcopt"]). A subprogram's events reach the same callable through its parent's context, so a workflow sees every level of a nested run. - The callable may be called from several threads at once (a list's calculations, a program's own threads), and should return quickly. An exception it raises is a warning; the calculation goes on.
- Store hits emit nothing. A list's events do not say which input they are from:
submiteach input with a callable of its own (functools.partial(show, i)) to tell them apart. - A program in a worker streams its events to the client while a message is in flight
(
ctx.progresson the wire, below); a program run without a listener hasctx.progressNone, so an unused progress channel costs nothing.
Stopping¶
runner.stop(future) (qccompute.stop for the default runner) asks a submitted
calculation to stop at its next step and keep what it reached: Stop & analyze in a live
view, a search ended on a budget. It is not a cancel, which discards the calculation.
- The program sees the request as
ctx.stop_requested(), checks it where it can end cleanly (an optimizer: before each step), and ends there: its output is a failure of kindstoppedwith its partial results (a search's frames so far), raised as for any failure.future.result()raisesqcdata.ProgramErrorwith that output. - The request covers the whole calculation: its subprograms see it through their
contexts, and a program whose subprogram stopped stops too. When a program lets a
stopped subprogram's failure escape, the runner fails it with kind
stopped(notprogram), the subprogram's error ascause. - In a worker process the request reaches the program through a file named in the
message's context (
stop_file), which the client creates while the message is in flight: onestatper check, and never the wrong message. The worker is asked, not killed, and serves on. - A calculation asked before it starts stops at its first step; a program with no steps runs to its end. The runner never stores a stopped output, and a store never reuses one.
Failures¶
A failed calculation raises qcdata.ProgramError, from compute as from sessions. Its
error is an ErrorInfo (kind, message, an optional code, and for a failed
subprogram the subprogram's error as cause), and its output is the whole failed
output: partial results, logs, record, traceback. A program whose result is one output
raises ProgramError(out.error, output=out); when a program raises without one, the
runner attaches the output it built. Any other exception from a program is a bug: the
runner reports it as kind internal, caused by the original exception.
The runner turns failures into failed outputs only where it needs data:
- a list runs every calculation, then raises
qcdata.ProgramErrors, anExceptionGroupof theProgramErrors whoseresultsholds every output in input order; - a session's
compute_manyruns every request, then raises the same group, whoseresultsholds every frame in request order (ProgramErrors.check(frames)); compute_batchreturns failures in place, a refused input included;raise_exc=Falsereturns failed outputs in place; the store and workers carry failed outputs as data.
A subprogram's failure that a program lets escape fails it with kind program, the
subprogram's name and its error as cause; a program that catches it sees the
subprogram's own error (kind stopped when the subprogram stopped on request). The
runner sets the kinds timeout (a deadline) and environment (a program not
installed, a worker that died: code worker_died).
Sessions¶
The same program and molecule at many geometries, warm-started: what optimizers, IRCs,
dynamics, path methods and finite differences need. prog.Session(inp) is both the
program's own API and the protocol object; open_session returns it.
with program.Session(inp) as s: # every calctype the method supports
f = s.compute() # the input's geometry and calctype
f = s.compute(x2) # warm start from this lineage's history
h = s.compute(x2, calctype="hessian") # energy and gradient too
g = s.compute(y, lineage=("irc", "forward"), seed=["ts"])
fd = s.compute_many([Request(positions=p, commit=False) for p in displaced])
- The input pins atoms, charge, multiplicity, model and keywords; its geometry and
calctype are a request's defaults.
calctypes=narrows what may run; a setting that applies to none of them is an input error (not_applicable). - Results are
Frames, with the sharedwarm_startrecord key. - A lineage is any JSON value, keyed by its canonical text (
qcdata.lineage_key).seedlists lineages to start from when the request's own has no history;commit(default true) makes a success the lineage's history. compute_manyruns requests of one lineage in order, each reading its lineage's history as it stands at its place; requests of different lineages may run concurrently, sharing the session's thread count (ctx.threadswhen it opened): the program gives each the count divided by those it runs at once.- A failed request raises
ProgramErrorwith its failed frame; the session and its other lineages go on, and the lineage's history is untouched. - Sessions may be called from several threads; the runner serializes those of programs that are not thread-safe.
qccompute.session(inp) hands back the program's own session in process, behind a thin
wrapper that counts each request's threads, so the numbers are exactly those of the
program used directly. A program in a worker gets a WorkerSession over the pipe, with
the same methods. A program without sessions gets a ComputeSession: one stateless
calculation per request (warm_start false), placed like any other calculation (in this
process, or in the program's workers). Its compute_many runs as a list does, sharing
the session's count: thirteen path images on eight threads run eight at a time. A
session holds no threads while it is idle; each request takes its count while it runs
(Threads).
The runner¶
Runner(threads=None, gpus="auto", isolation="auto") with submit (one input; returns
a concurrent.futures.Future of its output), compute (one input or a list) and
session; concurrent.futures.as_completed serves submitted futures. Every call takes
the same options:
store, timeout, rerun, tags, meta, raise_exc (not submit: its future
raises) and progress. qccompute.compute, qccompute.submit and qccompute.session
are the methods of the default runner: this machine (every core this process may use,
or OMP_NUM_THREADS, and every visible GPU), made on first use, never reconfigured.
submit is for workflows whose next calculation depends on the last (per reaction, a
transition-state search, the IRC from its saddle, then both ends), many at once with no
stage barriers. A stored output, or an input its program refuses, resolves the future
before submit returns; the others are planned at once and start in the order
submitted as the runner's threads and GPUs allow, sharing them with compute calls and
sessions (Threads). future.cancel() cancels a calculation whether it waits or
runs: one that waits never starts; one running in a worker process is killed with every
process it started (below, Workers); one running in this process ends cooperatively,
as for a timeout: ctx.timeout reads 0 for it and every subprogram, and it starts no
more subprograms. A cancelled calculation's output is discarded, never stored.
Every input goes one path: find the program, plan it (an input its program refuses
fails without running), place it (this process, or a worker process for programs with
in_process=False and for isolation="process"), call compute or compute_batch
with a context, record the execution.
Execution. The runner sets each output's execution: wall_time (the call's),
identities, threads when the program left it unset (the threads granted), and the
environment: hostname, host_cpu, host_mem_gib, cpu_model (as the operating
system names the CPU), libc (glibc 2.39) and devices (the granted GPUs' names, as
nvidia-smi gives them), each read once per process and filled only where the program
left it unset, so a program that knows better (one that ran elsewhere) sets its own. A
program granted a GPU that ran on none (a CPU fallback) says devices=[], which stays; a
worker fills its program's devices the same way before the output crosses the pipe.
The outputs of a batch ran together: each keeps a wall_time its program set (a
lockstep batch knows when each task finished), else gets the batch's, and the batch's
wall time is always timings["batch"]. One whose program left threads unset gets its
own input's part of the batch's threads: its own, scaled down when the batch was capped
(max(1, asked × granted // total asked)). So wall_time × threads
adds up across a batch's outputs, and a store's runs, to what the batch cost, to
rounding.
Resources. qc* never allocates machines or decides what runs when: a dispatcher (a batch system, Kubernetes) gives the process its cores, memory and GPUs, and workflows choose what to run. A runner gives each calculation one thread count and never runs more than its own count at once; whatever holds a count (the runner, a running calculation, a session) never has more than it running under it. Threads has the rules and the recipes. GPUs are whole: each goes to one calculation at a time. The inputs of a list that go to a program with batches are one calculation of the list, with the sum of their threads and the most GPUs any of them asks for; the program divides them.
Memory. There are no memory estimates and no memory scheduling: a plan's
memory_gib and device_memory_gib are informational, recorded and never scheduled.
What the runner grants, in ctx:
- Host memory. A calculation's share of its caller's: of the runner's for a top-level calculation or session (its threads' share or its GPUs', whichever is larger), of its parent's for a subprogram, of its session's for a request.
- GPU memory. Each granted GPU's, at most a subprogram's parent's.
- Keywords. The input's
memory_gibordevice_memory_gibreplaces either, at most what its caller has. A batch's singlectxgets the shares, not its inputs' keywords. Likethreads, these resource keywords are what a calculation asks for, and they are left out of its key. - Nothing is rounded: a keyword reaches the program as given, a share as computed, so a small grant is never zero and a keyword at the grant is never "capped" by rounding.
Placement. A program that may share the process runs in its threads; one that is not
thread-safe takes turns on a lock per program. The first time one of its calculations
waits for its turn (threads calling compute, futures, a list), the runner warns once
(RuntimeWarning) that they run one at a time and that isolation="process" runs them
side by side. Worker processes are kept for the next
call until the runner closes, in pools. A worker sees exactly the GPUs granted to it
(CUDA_VISIBLE_DEVICES, empty for none). Workers are pooled by program and GPUs,
whatever the threads of the calls they serve, because a program's state (a model
loaded, a GPU context) is costly to repeat: a list's batches of different sizes, and
calls with different threads, share the program's workers rather than starting one
each. Each call's threads reach the program in ctx.threads, which it applies to that
call (a subprocess's OMP_NUM_THREADS, torch.set_num_threads); a worker's numerical
libraries start with the threads of the call that started it (OMP_NUM_THREADS and the
others, unless set), which sizes the pools they make when they load; each call's pools
are then held at its own threads, as in process (Thread pools). On GPUs one
worker serves every calculation and session of a program on those GPUs, whose work
queues on the device anyway; without GPUs, at most as many as fit side by side in the
runner's threads at a call's threads. A call or session takes an idle
worker of its pool, else starts one while the pool has room, else shares the worker with
the fewest users, taking turns on it. A worker that dies or is killed for a timeout ends
the sessions it hosts.
Thread pools. Multithreaded BLAS gives different last bits at different thread counts,
so the runner holds the numerical libraries loaded in a process (OpenBLAS, MKL, BLIS,
FlexiBLAS and the OpenMP runtimes, as threadpoolctl finds them) at a calculation's
threads around each call of its program: compute, compute_batch, a session's opening
and each of its requests, in process and in workers alike. A calculation run alone thus
gives the same numbers in process and in a worker, whatever the caller's environment and
whichever call started a pooled worker.
- A count that belongs to the calling thread (OpenMP's, MKL's, OpenBLAS's on OpenMP) is set on the thread that calls the program and restored after.
- A count that belongs to the process (OpenBLAS on its own threads, BLIS, FlexiBLAS) is
set, and on return the call gives back what it found: a subprogram holds its own
threads and its parent gets its own back. In process, calculations running at once
with different threads share these counts (the one started last holds them); for
bitwise reproducibility across mixed thread counts, use
isolation="process". Once no calculation runs, every count is back where the caller had it. - A calculation that runs subprograms (its input names some) holds its own pools at one thread, and its count goes to its subprograms, which hold theirs while they run. So the threads in use, children included, stay within the count: a parent's pools at the whole count kept threads busy beside its children, because an idle OpenBLAS thread spins for about 0.1 s after each call. On 8 threads, a qcopt optimizer's own steps woke 7 such threads between gradients that axtb computed on all 8: 15-16 runnable threads on 8 cores, and 38 submitted TS searches, IRCs and end minimizations took 80-140 s instead of 24-26 s.
- A parent may raise its own pools up to its count for heavy linear algebra of its own, only while none of its subprograms runs, lowering them again before the next one starts and before it returns.
- A session's requests are held at its count, in process (the runner's wrapper around the program's own session) and in workers.
The store¶
A store files runs by two identities, kept apart:
- which calculation: the key,
qcdata.calculation_key(inp), the input as given; - which code produced the numbers: the run's identities,
{program: numerics}for the input's program and every subprogram it names, recursively. A calculation runs no program its input does not name (ctxrefuses it), so these are every program that made the numbers. A program named with several specs (g-xTB gradients with GFN2 Hessians, one program) has its distinct identities sorted and joined by;("gfn2:5;gxtb-2.0.1:7").givenresults add no identities: they are data in the input, so already in the key.
With compute(..., store=store) the runner computes the input's identities from the
programs it would run now, looks the key up with them, and returns a stored run only if
it was made with exactly those identities: a success always, a failure only when the
input determines it (kind input, unsupported or convergence, or program with such
a cause; never resource, backend, environment, timeout, numerical or
internal). Misses run, and every run is stored with its identities, except a stopped
one (kind stopped, or failed because a subprogram stopped): it is where the caller
chose to end the run, not what the input gives. Every output the runner returns carries the identities of the programs
that made it in execution.identities, with or without a store, so an output read back
says which code made it; a stored run keeps the ones it was made with. rerun=True
always runs; tags go onto every key, cache hits included;
meta goes with each run. An output whose record says warm_start true depends on
history, not only on its input, and is never stored. Every output a runner returns with
a store names its run, execution.store_run = "<key>/<run>": a hit as the store marks
it, a fresh run once it is stored. So a result is cited exactly, and read back later
with qcstore.Store.get(key, run=...) even after a newer run became the preferred one.
Concurrent misses of one key each compute, and the store keeps every run.
The runner needs four methods of a store (qccompute.runner.Store): key(inp),
get(key, identities), put(output, tags=, meta=) (which files the run under
output.execution.identities) and tag(key, *tags).
Workers¶
python -m qccompute.serve PROGRAM hosts one program's calculations and sessions in its
own process, so a crash ends the worker and not its caller, and a program keeps its state
between calls. qccompute.worker.Worker is its client.
-
Which program:
PROGRAMis an installed program's name, amodule:attribute, orpickle:PATH. A runner names an installed program (the object its entry point names) by its name, a registered one by the module attribute that holds it, and any other registered instance (one made in a script) by a pickle of it, written once per process to a private directory. A pickle stores classes and functions by reference, so they must be importable in the worker; one that holds a lambda, a closure, or anything defined in the script being run cannot be rebuilt there, and running it in a worker fails with kindenvironment, saying so. -
Transport: stdin and stdout, one JSON message per line, one client per connection;
--socket PATHserves a Unix socket on the same framing.--scratch DIR: the calculations' temporary scratch directories are made inDIR, which aWorkermakes for its worker and removes when it closes it, killed or not. - Hello: the first line,
{"protocol": 1, "program", "version", "capabilities", "array_files": true, "calls": true}. - Envelope: every message has an
id; replies are{"id", "ok": true, "result"}or{"id", "ok": false, "error"}.ok: falseis for the message itself; a failed calculation is a result (a failed output or frame). - Ops:
open,compute(a session's request, or a one-offinput),compute_many,compute_batch,reset,plan,close,lineages,options. End of input closes every session. - Options: a client turns on what the hello offers with
options {array_dir, calls}, per connection; a plain JSON-lines client sends nothing and gets neither. - Large arrays: with
array_dir(workers are local), an array field over 1 MiB, in a message either way, crosses as a.npyfile in that directory, with qcdata's pointer{path, shape, dtype, sha256}in its place, as in a saved model; the reader checks it and deletes the file. A 1200 x 1200 Hessian's output crosses in 20 ms instead of 0.85 s and a 28 MB line. - Subprograms run in the worker when they can, else in the client: with
calls: true, a subprogram call of the worker's program (ctx.compute,ctx.session) runs in the worker itself, under the calculation's thread count, when it can run there exactly as the client would run it: every program its input names is found in the worker with the identities the client gave for the calculation (the wire context'sidentities), may run in process, and its plan uses no GPU. So qcopt's axtb gradients cost no round trip: one caffeine optimization took 10.9 ms per gradient instead of 12.8-15.0 through the client, and 16 submitted at once on 8 threads 1.0-1.2 s instead of 1.5-2.0 s, with identical numbers. Every other call comes back to the client as a call line,{"call", "op", ...}with the ops above, which the client answers in any order ({"call", "ok", "result"}) while the message is in flight: a program registered only in the client, one that needs a worker of its own, a GPU calculation; such a call takes its threads from the calculation's count in the worker before it goes. A nested worker is started by the client, never by a worker. Either way the count, the time left, a stop, progress (paths compose: the worker's["axtb"]under the client's["qcopt"]) and the check that the input names the program apply alike. Withoutcalls, subprograms run in the worker's own default runner. A list of subprograms runs in the worker only if all of it can. - Context on the wire:
{threads, memory_gib, device_memory_gib, timeout, scratch_dir, progress, stop_file, identities}; the GPUs are fixed when the worker starts.identitiesare those of the programs the client would run for the calculation ({program: numerics}), against which the worker checks its own before running a subprogram itself.stop_filenames a file whose existence asks the calculation to stop (ctx.stop_requested()checks it): the client creates it forrunner.stop, while the message is in flight, so a request never reaches another message. Withprogress: true(the client's caller listens) the program's progress events stream back as lines{"id", "progress": event}naming the message in flight (for a session, its request's message); the client hands each to the caller's callable. - Numbers: shortest round-trip decimals, so positions and results cross bit for bit; no NaN or infinity on the wire.
- Hygiene: stdout carries protocol lines only; the worker points file descriptor 1 at stderr, because native libraries print.
- Crashes: a dead worker is kind
environment, codeworker_died, with the exit status and the end of its stderr. After a GPU fault that poisons the device context (codedevice_lost) the worker answers, then exits 3. A runner kills a worker whose calculation exceeds its timeout or is cancelled, with every process it started, however deep and in whatever session (qccodec runs each external program in a session of its own): the tree is found through/proc, frozen, then killed. Closing a runner kills a worker still running a calculation rather than waiting for it.
External programs¶
qccodec registers each external program as a program (qccodec.CodecProgram). An adapter
encodes the input, runs the command in the context's scratch directory with its output
captured and the context's deadline, decodes what it wrote, and raises ProgramError
with the failed output and the logs when it fails.
- An input that requests threads (
threads, or the program's own: ORCApal, Psi4-n, CRESTthreads) runs with its request; one that requests none runs with the calculation's count (xtb with 1, its own cap), through the program's own setting orOMP_NUM_THREADSand the other numerical libraries' variables. A GPU program sees its granted GPUs inCUDA_VISIBLE_DEVICES. - Codecs that call a Python library take over the process's output streams and working
directory while they run, so they are
in_process=Falseand run in workers. - geomeTRIC and Sella run their gradient subprograms through
ctx.compute, which from a worker comes back to the runner's process. - An external program's
numericsis the sha256 of the executable that would run; a library's, its version. - A
qcdata.FileInputruns under thefilesprogram and returns every file the command wrote.
Cost¶
On one core, qccompute.compute adds about 10 µs to a direct call of a FunctionProgram
with a version (finding it, planning it, its context, recording the execution and the
identities, and about 2.4 µs of it holding numpy's OpenBLAS at the calculation's
threads and giving the count back); a FunctionProgram without a
version adds about 6 µs more, the digest of its functions that is its identity. The
runner hands its plan on in ctx.plan, so a program need not plan twice. Through qccodec, a trivial external
command takes about 0.26 ms more than launching it in a fresh directory by hand. A worker
round trip for the toy program costs about 0.14 ms (JSON on a pipe), and starting a
worker about 0.15 s plus the program's own imports, once.
Conformance¶
qccompute.testing.check_program(program, inputs) runs a program's own small inputs and
checks the contract: schemas that validate, plans that reproduce themselves from
resolved.settings, unknown keys refused with kind input by plan and compute, the
thread count honoured, standard fields in atomic-unit shapes, energy components that add
up, and for sessions and batches, agreement with plain computes, refusal of other
calctypes, and a session that survives a failed request.