QCFractalCompute - Compute Managers and Workers#

QCFractal executes quantum chemical calculations via compute managers deployed to resources suitable for these types of calculations. This document illustrates how to set up and run a compute manager on HPC resources.

Compute manager setup for an HPC cluster#

Note

The instructions in this section are performed on the head/login node of your HPC cluster. By the end, you will start up a compute manager on the head node, and it will launch jobs on your behalf. If your cluster admin forbits long-running processes on the head node, then refer to Execution without interfacing with an HPC scheduler.

Install the base environment for the compute manager, using mamba, and activate it:

$ mamba create -n qcfractalcompute -c conda-forge qcfractalcompute
$ mamba activate qcfractalcompute

This will create the conda environment qcfractalcompute, which the compute manager will run under.

Next, create the manager config as qcfractal-manager-config.yml, using the content below as your starting point. Fill in the components given in <brackets>. For this example, we are assuming a cluster using LSF as the scheduler; for other scheduler types, see Configuration for different HPC schedulers:

# qcfractal-manager-config.yml
---
cluster: <cluster_name>           # descriptive name to present to QCFractal server
loglevel: INFO
logfile: qcfractal-manager.log
update_frequency: 60.0

server:
  fractal_uri: <fractal_url>      # e.g. https://qcarchive.molssi.org
  username: <compute_identity>
  password: <compute_key>
  verify: True

executors:
  cpuqueue:
    type: lsf
    workers_per_node: 4           # max number of workers to spawn per node
    cores_per_worker: 16          # cores per worker
    memory_per_worker: 96         # memory per worker, in GiB
    max_nodes: 2                  # max number of jobs to have queued/running at a time
    walltime: "4:00:00"           # walltime, given in `hours:min:seconds`, alternatively in integer minutes
    project: null
    queue: <queue_name>           # name of queue to launch to
    request_by_nodes: false
    scheduler_options:            # add additional options for submission command
      - "-R fscratch"
    compute_tags:                 # only claim tasks with these tags; '*' means all tags accepted
      - '*'
    environments:
      use_manager_environment: False   # don't use the manager environment for task execution
      conda:
        - <worker_conda_env_name>      # name of conda env used for task execution; see below for example
    worker_init:
      - source <absolute_path>/worker-init.sh   # initialization script for worker; see below for example

Create the file worker-init.sh; this is run before worker startup:

# worker-init.sh
#!/bin/bash

# Make sure to run bashrc
source $HOME/.bashrc

# Don't limit stack size
ulimit -s unlimited

# make scratch space
# CUSTOMIZE FOR YOUR CLUSTER
mkdir -p /fscratch/${USER}/${LSB_JOBID}
cd /fscratch/${USER}/${LSB_JOBID}

# Activate qcfractalcompute conda env
conda activate qcfractalcompute

Set the absolute path to this file in the line - source <absolute_path>/worker-init.sh in qcfractal-manager-config.yml above.

We will also create a conda environment used by each worker to execute tasks, e.g. an environment suitable for using psi4, with a conda environment file such as worker-env.yml:

# worker-env.yml
---
name: qcfractal-worker-psi4-18.1
channels:
  - conda-forge/label/libint_dev
  - conda-forge
  - defaults
dependencies:
  - python =3.10
  - pip
  - qcengine
  - psi4 =1.8.1
  - dftd3-python
  - gcp-correction
  - geometric
  - scipy

  - pip:
    - basis_set_exchange

And creating a conda environment from it with mamba:

$ mamba env create -f worker-env.yml

Set the name of this conda env (qcfractal-worker-psi4-18.1) in the line - <worker_conda_env_name> in qcfractal-manager-config.yml above.

Finally, start up the compute manager:

$ qcfractal-compute-manager --config qcfractal-manager-config.yml

The compute manager will read its config file, communicate with the QCFractal server to claim tasks, and launch jobs to the HPC scheduler as needed to execute those tasks using the worker conda environment. To keep it running beyond your current session if connected via SSH, consider running the compute manager under tmux or screen.

Configuration for different HPC schedulers#

HPC cluster schedulers vary in behavior, so you will need to adapt your qcfractal-manager-config.yml to the scheduler of the HPC cluster you intend to use. The configuration keys available for each type of record in the executors list are referenced here.

Every executor, whatever its type, accepts the keys on ExecutorConfig below - including compute_tags, cores_per_worker, memory_per_worker, worker_init, and the nested environments block. The scheduler-specific classes that follow add their own keys on top.

class ExecutorConfig[source]#

Bases: QCFComputeConfigBase

Fields#
Field	Type	Required	Default
`bind_address`	`str` \| `None`	No	`None`
`compute_tags`	`list`[`str`]	Yes
`cores_per_worker`	`int`	Yes
`environments`	`PackageEnvironmentSettings`	No	factory
`extra_executor_options`	`dict`[`str`, `Any`]	No	`{}`
`memory_per_worker`	`float`	Yes
`scratch_directory`	`str` \| `None`	No	`None`
`type`	`str`	Yes
`worker_init`	`list`[`str`]	No	`[]`

type: str#

compute_tags: list[str]#

worker_init: list[str]#

scratch_directory: str | None#

bind_address: str | None#

cores_per_worker: int#

memory_per_worker: float#

extra_executor_options: dict[str, Any]#

environments: PackageEnvironmentSettings#

class PackageEnvironmentSettings[source]#

Bases: QCFComputeConfigBase

Environments with installed packages that can be used to run calculations

The compute manager will query these environments to see what packages are installed, and direct appropriate calculations to them.

Fields#
Field	Type	Required	Default
`apptainer`	`list`[`str`]	No	`[]`
`conda`	`list`[`str`]	No	`[]`
`use_manager_environment`	`bool`	No	`True`

use_manager_environment: bool#: Use the environment that the manager is running in for computation. May be in addition to other environments

conda: list[str]#: List of conda environments to query for installed packages

apptainer: list[str]#: List of paths to apptainer/singularity files to query for installed packages

class SlurmExecutorConfig[source]#

Bases: ExecutorConfig

Fields#
Field	Type	Required	Default
`account`	`str` \| `None`	No	`None`
`bind_address`	`str` \| `None`	No	`None`
`compute_tags`	`list`[`str`]	Yes
`cores_per_worker`	`int`	Yes
`environments`	`PackageEnvironmentSettings`	No	factory
`exclusive`	`bool`	No	`True`
`extra_executor_options`	`dict`[`str`, `Any`]	No	`{}`
`max_nodes`	`int`	Yes
`memory_per_worker`	`float`	Yes
`partition`	`str` \| `None`	No	`None`
`scheduler_options`	`list`[`str`]	No	`[]`
`scratch_directory`	`str` \| `None`	No	`None`
`type`	`Literal`[‘slurm’]	No	`'slurm'`
`walltime`	`str`	Yes
`worker_init`	`list`[`str`]	No	`[]`
`workers_per_node`	`int`	Yes

type: Literal['slurm']#

walltime: Annotated[str, BeforeValidator(func=_walltime_must_be_str, json_schema_input_type=PydanticUndefined)]#

exclusive: bool#

partition: str | None#

account: str | None#

workers_per_node: int#

max_nodes: int#

scheduler_options: list[str]#

compute_tags#

worker_init#

scratch_directory#

bind_address#

cores_per_worker#

memory_per_worker#

extra_executor_options#

environments#

class TorqueExecutorConfig[source]#

Bases: ExecutorConfig

Fields#
Field	Type	Required	Default
`account`	`str` \| `None`	No	`None`
`bind_address`	`str` \| `None`	No	`None`
`compute_tags`	`list`[`str`]	Yes
`cores_per_worker`	`int`	Yes
`environments`	`PackageEnvironmentSettings`	No	factory
`extra_executor_options`	`dict`[`str`, `Any`]	No	`{}`
`max_nodes`	`int`	Yes
`memory_per_worker`	`float`	Yes
`queue`	`str` \| `None`	No	`None`
`scheduler_options`	`list`[`str`]	No	`[]`
`scratch_directory`	`str` \| `None`	No	`None`
`type`	`Literal`[‘torque’]	No	`'torque'`
`walltime`	`str`	Yes
`worker_init`	`list`[`str`]	No	`[]`
`workers_per_node`	`int`	Yes

type: Literal['torque']#

walltime: Annotated[str, BeforeValidator(func=_walltime_must_be_str, json_schema_input_type=PydanticUndefined)]#

account: str | None#

queue: str | None#

workers_per_node: int#

max_nodes: int#

scheduler_options: list[str]#

compute_tags#

worker_init#

scratch_directory#

bind_address#

cores_per_worker#

memory_per_worker#

extra_executor_options#

environments#

class LSFExecutorConfig[source]#

Bases: ExecutorConfig

Fields#
Field	Type	Required	Default
`bind_address`	`str` \| `None`	No	`None`
`bsub_redirection`	`bool`	No	`True`
`compute_tags`	`list`[`str`]	Yes
`cores_per_worker`	`int`	Yes
`environments`	`PackageEnvironmentSettings`	No	factory
`extra_executor_options`	`dict`[`str`, `Any`]	No	`{}`
`max_nodes`	`int`	Yes
`memory_per_worker`	`float`	Yes
`project`	`str` \| `None`	No	`None`
`queue`	`str` \| `None`	No	`None`
`request_by_nodes`	`bool`	No	`True`
`scheduler_options`	`list`[`str`]	No	`[]`
`scratch_directory`	`str` \| `None`	No	`None`
`type`	`Literal`[‘lsf’]	No	`'lsf'`
`walltime`	`str`	Yes
`worker_init`	`list`[`str`]	No	`[]`
`workers_per_node`	`int`	Yes

type: Literal['lsf']#

walltime: Annotated[str, BeforeValidator(func=_walltime_must_be_str, json_schema_input_type=PydanticUndefined)]#

project: str | None#

queue: str | None#

workers_per_node: int#

max_nodes: int#

request_by_nodes: bool#

bsub_redirection: bool#

scheduler_options: list[str]#

compute_tags#

worker_init#

scratch_directory#

bind_address#

cores_per_worker#

memory_per_worker#

extra_executor_options#

environments#

Execution without interfacing with an HPC scheduler#

When running with a configuration like that above, the compute manager must remain alive on the head/login node of the cluster in order to execute tasks. If leaving a long-running process running on the head node is undesirable, then consider using a local executor configuration instead, replacing the executors section in qcfractal-manager-config.yml with e.g.:

executors:
  local_executor:
    type: local
    max_workers: 4                # max number of workers to spawn
    cores_per_worker: 16          # cores per worker
    memory_per_worker: 96         # memory per worker, in GiB
    compute_tags:
      - '*'
    environments:
      use_manager_environment: False
      conda:
        - <worker_conda_env_name>      # name of conda env used by worker; see below for example
    worker_init:
      - source <absolute_path>/worker_init.sh

You will then need to create a submission script suitable for your HPC scheduler that requests the appropriate resources, activates the qcfractalcompute conda environment, and runs qcfractal-compute-manager --config qcfractal-manager-config.yml itself. You can then manually submit jobs using this script as needed to complete tasks available on the QCFractal server.

Using the local executor type is also recommended for running a compute manager on a standalone host, or within a container on e.g. a Kubernetes cluster.

class LocalExecutorConfig[source]#

Bases: ExecutorConfig

Fields#
Field	Type	Required	Default
`bind_address`	`str` \| `None`	No	`None`
`compute_tags`	`list`[`str`]	Yes
`cores_per_worker`	`int`	Yes
`environments`	`PackageEnvironmentSettings`	No	factory
`extra_executor_options`	`dict`[`str`, `Any`]	No	`{}`
`max_workers`	`int`	Yes
`memory_per_worker`	`float`	Yes
`scratch_directory`	`str` \| `None`	No	`None`
`type`	`Literal`[‘local’]	No	`'local'`
`worker_init`	`list`[`str`]	No	`[]`

type: Literal['local']#

max_workers: int#

compute_tags#

worker_init#

scratch_directory#

bind_address#

cores_per_worker#

memory_per_worker#

extra_executor_options#

environments#