Many calculations at once¶
Pass compute a list of inputs to get a list of outputs in the same order. The inputs may
mix programs, methods and calculation types.
import qccompute
import qcdata
from qcdata import CalcType, Model, ProgramInput, Structure
inputs = [
ProgramInput(
program="xtb",
calctype=CalcType.gradient,
structure=structure,
model=Model(method="gfn2"),
keywords={"threads": 2}, # two threads each
)
for structure in structures
]
outputs = qccompute.compute(inputs)
How a list runs¶
- Lookup. With a
store, every input is looked up first; reusable runs come back without running. - Planning. Each program plans its inputs (
program.plan(inp)): an input it refuses fails at once, without running, and the plan says which GPUs the calculation needs. The plan is handed to the program, so nothing is planned twice. - Sharing. The inputs that run share the runner's thread count: each gets the count
divided by how many run, at least 1, unless it sets
threads; they start in input order, as threads free up (Threads). On 16 threads, eight 2-thread xtb calculations run at once. - Placement. Programs that may share this process run in its threads (one at a time
for programs that are not thread-safe); programs that cannot (they capture the
process's output streams, as Python libraries run by qccodec do) run in worker
processes, which stay alive for the next call. A program that evaluates many inputs at
once (
compute_batch, machine-learned potentials, qcopt) gets all of its inputs together in one call, with their threads together, and divides them among its inputs.
Failures¶
Every calculation runs; then, if any failed, qcdata.ProgramErrors is raised. It is an
ExceptionGroup of qcdata.ProgramErrors, each with its failed output, and its
results holds every output in input order:
try:
outputs = qccompute.compute(inputs)
except qcdata.ProgramErrors as errors:
outputs = errors.results
failed = [o for o in outputs if not o.success]
# or handle failures by kind
try:
outputs = qccompute.compute(inputs)
except* qcdata.ProgramError as group:
...
raise_exc=False returns failed outputs in place.
As they finish¶
Submit each input and take the futures as they complete. Give each threads (usually
1), or they run one at a time:
from concurrent.futures import as_completed
with qccompute.Runner(threads=32) as runner:
one = [i.model_copy(update={"keywords": {**i.keywords, "threads": 1}}) for i in inputs]
for future in as_completed([runner.submit(i, timeout=600) for i in one]):
out = future.result() # raises qcdata.ProgramError for a failure
print(out.input_data.program, out.results.energy)
When the next calculations depend on earlier ones, submit each as its inputs become known and get a future of its output: see Dynamic workflows.