Skip to content

qccompute

qccompute runs programs on qcdata inputs. Every program runs the same way, whether it implements qccompute's protocol in Python or is an external program that qccodec brings in:

python -m pip install "qccompute[qccodec]" qcstore

qccompute alone is the runner, for programs that implement its protocol (pip install qccompute); the qccodec extra adds external programs (xtb, ORCA, Psi4, ...) and qcstore is the calculation store. Coming from 0.22? See the migration guide.

One calculation

import qccompute
from qcdata import CalcType, Model, ProgramInput, Structure

water = Structure(
    symbols=["O", "H", "H"],
    geometry=[[0.0, 0.0, 0.0], [0.524, 1.687, 0.48], [1.146, -0.45, -1.354]],  # bohr
)
inp = ProgramInput(
    program="xtb", calctype=CalcType.gradient, structure=water, model=Model(method="gfn2")
)
out = qccompute.compute(inp)
print(out.results.energy, out.results.gradient)

qccompute finds the program by name (ProgramInput.program) among the installed programs (the qccompute.programs entry points) and those registered in this process with qccompute.register_program. It hands the calculation its thread count (the runner's whole count unless the input sets threads; see Threads), runs it, and records the run in out.execution (wall time, host, threads).

Failures raise

A failed calculation raises qcdata.ProgramError. Its error says what kind of failure it was (input, convergence, timeout, environment, ...) and its output is the failed ProgramOutput, with the partial results, the logs and the traceback:

from qcdata import ProgramError

try:
    out = qccompute.compute(inp)
except ProgramError as e:
    print(e.error.kind, e.error.message)
    print(e.output.logs)

raise_exc=False returns the failed output instead. A program's bug (any other exception) is a failure of kind internal, caused by the original exception.

Many calculations

A list runs side by side and returns its outputs in input order. The inputs may mix programs, methods and calculation types:

outputs = qccompute.compute([inp1, inp2, inp3])
  • Every input is planned first: an input its program refuses (an unknown keyword) fails at once, without running.
  • They share the runner's threads: each gets the count divided by the inputs that run (Threads). A program that evaluates many inputs at once (a machine-learned potential) gets its inputs together.
  • Everything runs, then the failures are raised together as qcdata.ProgramErrors (an ExceptionGroup of ProgramErrors). Its results holds every output in input order, so no result is lost:
import qcdata

try:
    outputs = qccompute.compute(inputs)
except qcdata.ProgramErrors as errors:
    outputs = errors.results

For outputs as they finish, submit each input and use concurrent.futures.as_completed (see Dynamic workflows).

Sessions: one molecule at many geometries

Optimizers, IRCs, dynamics and finite differences evaluate one molecule at many geometries. A session pins the program, model, keywords, atoms, charge and multiplicity, and lets a program keep what speeds the next step (orbitals for a warm start):

with qccompute.session(inp) as s:
    frame = s.compute()                       # the input's own geometry and calctype
    frame = s.compute(new_positions)          # warm-started from the last success
    hessian = s.compute(new_positions, "hessian")

Only a program with sessions of its own (capabilities().sessions) warm-starts: you get its own session, behind a thin wrapper that counts each request's threads, so the numbers are exactly those of the program used directly. Any other program (most external programs, tblite, a FunctionProgram) gets a session over compute, which computes every request from scratch (record["warm_start"] is false): a convenience, not a speed-up. A failed request raises ProgramError and the session stays usable; compute_many runs every request, then raises qcdata.ProgramErrors, whose results holds every frame in request order. A session has one thread count (qccompute.session(inp, threads=8), else as for a calculation), which its requests share and hold only while they run (Threads). Lineages name separate histories in one session (both branches of an IRC), seed starts one from another, and commit=False reads a history without moving it; see Sessions, lineages and seeds.

A share of the machine: the Runner

qccompute.compute and qccompute.session are the methods of the default runner: every core this process may use (its CPU affinity and cgroup quota, or OMP_NUM_THREADS if set) and every visible GPU. It is made on first use and cannot be reconfigured.

A Runner of your own takes a share:

with qccompute.Runner(threads=16, gpus=["0"], isolation="process") as runner:
    outputs = runner.compute(inputs, timeout=3600)
  • threads: the runner's thread count, which its calculations share; it never runs more at once (Threads). gpus: the GPUs it may use, each by one calculation at a time; a calculation gets the GPUs its program's plan asks for, and ctx.device_ordinals() turns them into CUDA device numbers in the program's process. Each calculation gets its threads, GPUs and memory in a qccompute.RunContext.
  • Memory is never scheduled: a calculation gets its share of the runner's, and a memory_gib or device_memory_gib keyword replaces it (Threads).
  • isolation: "auto" runs programs in this process unless they cannot share it (they say in_process=False), which run in worker processes; "process" runs every program in a worker, so a crash ends only the worker.
  • Workers stay alive between calls, so a program keeps its state; close the runner (or use it as a context manager) to stop them. A program's workers serve its calls whatever their threads, so a model loads once: on a GPU one worker per GPU, which its calculations and sessions share; on the CPU as many as run side by side.

The runner never changes a number by itself. A grant may change a program's own choices (a storage scheme picked from the memory granted), and the program records each one in output.resolved.

A calculation store

outputs = qccompute.compute(inputs, store=store, tags=["screen-1"], meta={"by": "me"})

With a store, every input is looked up first and only misses run; every run is stored. A store files runs by the calculation's key (the input as given, qcdata.calculation_key) and by the identities of the programs that made them: {program: numerics} for the input's program and every subprogram it names (a program named with several specs has its identities joined by ;). Every output says which code made it in execution.identities, with or without a store. A stored run is reused only when its identities are those of the programs that would run now; a stored failure only when the input determines it (kind input, unsupported or convergence, or program with such a cause), never one of the machine or the moment (resource, timeout, environment, ...). A stopped run is never stored. rerun=True always runs, past a stored failure too; tags go onto every key, cache hits included. Every output returned with a store names its run, computed or hit, in execution.store_run ("<key>/<run>"), so a network or a report can cite the exact run and read it back with store.get(key, run=...). A store implements key, get, put and tag (qccompute.runner.Store); qcstore is one. Rerunning a workflow from a store is all cache hits only when it asks for the same inputs; a workflow whose choices depend on timing keeps its own decision state (see Dynamic workflows).

One calculation at a time, as futures

qccompute.submit(inp) (and Runner.submit) starts a calculation and returns a concurrent.futures.Future of its output, for workflows whose next step depends on the last: per reaction, the IRC starts as soon as its transition-state search ends, with no stage barriers. Submitted calculations start in the order submitted, as the runner's threads allow: set threads (usually 1) on each to run many at once (Threads). A stored result resolves at once, and future.cancel() ends a calculation even while it runs. See Dynamic workflows.

A program that is not thread-safe (qcopt) runs one calculation at a time per process. The runner warns the first time one waits for its turn; Runner(isolation="process") runs them side by side in worker processes.

Progress

Programs can report what they are doing while they run (qcopt: one event per step). Pass a callable to see it:

def show(event):  # {"event": "step", "step": 12, "energy": -76.4, "path": ["qcopt"]}
    print(event["path"], event)

qccompute.compute(inp, progress=show)

Each event is a small JSON object of the program's own, with path added: the program names from your input down to the one that emitted it, so a subprogram's events (["neb", "qcopt"]) arrive too. The callable may be called from several threads at once. Stored outputs emit nothing, and nothing is stored. A program in a worker process streams its events back. submit and session take progress= too; to tell a list's inputs apart, submit each with its own callable.

A submitted calculation can be stopped at its next step, keeping what it reached:

future = qccompute.submit(search, progress=show)
...
qccompute.stop(future)  # the program sees ctx.stop_requested() and ends there
try:
    future.result()
except qcdata.ProgramError as e:  # kind "stopped", with the frames reached so far
    frames = e.output.results.frames

A stopped output is never stored. future.cancel() instead ends the calculation and discards it.

Programs

A program is an object implementing qccompute.program.Program: its name, version(), capabilities(), the JSON Schema of its settings and of its record, plan(inp) (a dry run that validates and resolves every "auto") and compute(inp, ctx=None). Programs register an instance under the qccompute.programs entry points:

[project.entry-points."qccompute.programs"]
myprogram = "mypackage.qc:program"

The protocol is structural, so a program without subprograms imports only qcdata. A program with subprograms runs them through ctx.compute and ctx.session; called directly (ctx is None), it returns qccompute.compute(inp). See Programs and sessions and the program interface.

For models and scripts, FunctionProgram makes a program of plain functions:

"""Make a program of two Python functions and run it like any other program.

A Morse potential between two atoms: the smallest complete program. The runner gives
it lists, sessions and the store, exactly as any other program.
"""

import numpy as np
from qcdata import CalcType, ProgramInput, Structure

import qccompute
from qccompute.program import FunctionProgram

D, A, R0 = 0.17, 1.03, 1.40  # well depth (Eh), width (1/bohr), bond length (bohr)


def morse(x: np.ndarray) -> tuple[float, np.ndarray]:
    """Energy (Eh) and gradient (Eh/bohr) of a Morse bond at positions x (bohr)."""
    d = x[1] - x[0]
    r = float(np.linalg.norm(d))
    e = np.exp(-A * (r - R0))
    de_dr = 2 * D * A * e * (1 - e)
    return D * (1 - e) ** 2, np.array([-de_dr * d / r, de_dr * d / r])


qccompute.register_program(FunctionProgram("morse", morse))

h2 = Structure(
    symbols=["H", "H"], geometry=np.array([[0.0, 0.0, 0.0], [0.0, 0.0, 1.6]])
)
inp = ProgramInput(program="morse", calctype=CalcType.gradient, structure=h2)

out = qccompute.compute(inp)
print(out.results.energy, out.results.gradient[1])

# A scan along the bond, side by side: the input at each new geometry
scan = [
    ProgramInput.from_spec(inp, h2.with_geometry([[0, 0, 0], [0, 0, r]]))
    for r in np.linspace(1.0, 3.0, 9)
]
energies = [o.results.energy for o in qccompute.compute(scan)]
print(np.round(energies, 4))

qccompute.testing.check_program(program, inputs) checks a program against the contract (schemas, plans, typed errors, the thread count, sessions, batches); qccompute.testing.toy is a complete example program.

External programs

qccompute[qccodec] installs qccodec, which registers xtb, CREST, ORCA, Psi4, TeraChem, DFTB+, geomeTRIC, Sella, tblite and machine-learned potentials as programs. Their keywords are the programs' own; each program page explains the mapping. A program that is not on PATH is found with QCCODEC_<PROGRAM>_EXECUTABLE. A qcdata.FileInput (a command and its files) runs under qccodec's files program and returns every file the command wrote; see File Input.

Workers

python -m qccompute.serve PROGRAM hosts a program in its own process, over JSON lines on stdin and stdout (or a Unix socket); qccompute.worker.Worker is its client, with the same compute, plan and sessions as the program in process. The runner uses them for programs that cannot share a process and for isolation="process".

A program in a worker runs its subprograms back in your process: geomeTRIC's gradients can come from a FunctionProgram you registered in your script, and they share the calculation's thread count. Large arrays (Hessians, wavefunctions) cross between the processes as .npy files rather than as text.