Base Record Models

class Provenance

Bases: BaseModel

Provenance information.

Fields

Field	Type	Required	Default
creator	str	Yes
routine	str	No	''
version	str	No	''

creator: str

The name of the program, library, or person who created the object

version: str

The version of the creator, blank otherwise

routine: str

The name of the routine or function within the creator, blank otherwise

class PriorityEnum

Bases: int, Enum

The priority of a Task. Higher priority will be pulled first.

high = 2

normal = 1

low = 0

__init__(*args, **kwds)

class RecordStatusEnum

Bases: str, Enum

The state of a record object. The states which are available are a finite set.

complete = 'complete'

invalid = 'invalid'

running = 'running'

error = 'error'

waiting = 'waiting'

cancelled = 'cancelled'

deleted = 'deleted'

classmethod make_ordered_status(statuses)

Returns a list of the given statuses but in a defined order

Parameters:

statuses (Iterable[RecordStatusEnum])

Return type:

list[RecordStatusEnum]

__init__(*args, **kwds)

class OutputTypeEnum

Bases: str, Enum

What type of data is stored

stdout = 'stdout'

stderr = 'stderr'

error = 'error'

__init__(*args, **kwds)

class OutputStore

Bases: BaseModel

Storage of outputs and error messages, with optional compression

Fields

Field	Type	Required	Default	Alias
compression_type	CompressionEnum	No	<CompressionEnum.none: 'none'>
data_	Annotated[Any, BytesAnnotation] | None	No	None	data
output_type	OutputTypeEnum	Yes

output_type: OutputTypeEnum

The type of output this is (stdout, error, etc)

compression_type: CompressionEnum

Compression method (such as lzma)

data_: QCPortalBytes | None

propagate_client(client, history_base_url)

property data: Any

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

class ComputeHistory

Bases: BaseModel

Fields

Field	Type	Required	Default	Alias
id	int	Yes
manager_name	str | None	Yes
modified_on	datetime	Yes
outputs_	dict[str, OutputStore] | None	No	None	outputs
provenance	Provenance | None	Yes
record_id	int	Yes
status	RecordStatusEnum	Yes

id: int

record_id: int

status: RecordStatusEnum

manager_name: str | None

modified_on: datetime

provenance: Provenance | None

outputs_: dict[str, OutputStore] | None

propagate_client(client, record_base_url)

fetch_all()

property outputs: dict[str, OutputStore]

get_output(output_type)

Parameters:

output_type (OutputTypeEnum)

Return type:

Any

property stdout: Any

property stderr: Any

property error: Any

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

class NativeFile

Bases: BaseModel

Storage of native files, with compression

Fields

Field	Type	Required	Default	Alias
compression_type	CompressionEnum	Yes
data_	Annotated[Any, BytesAnnotation] | None	No	None	data
name	str	Yes

name: str

Name of the file

compression_type: CompressionEnum

Compression method (such as lzma)

data_: QCPortalBytes | None

propagate_client(client, record_base_url)

fetch_all()

property data: Any

save_file(directory, new_name=None, keep_compressed=False, overwrite=False)

Saves the file to the given directory

Parameters:

• directory (str)

• new_name (str | None)

• keep_compressed (bool)

• overwrite (bool)

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

class RecordComment

Bases: BaseModel

Fields

Field	Type	Required	Default
comment	str	Yes
id	int	Yes
record_id	int	Yes
timestamp	datetime	Yes
username	str | None	Yes

id: int

record_id: int

username: str | None

timestamp: datetime

comment: str

class RecordTask

Bases: BaseModel

Fields

Field	Type	Required	Default
compute_priority	PriorityEnum	Yes
compute_tag	str	Yes
function	str | None	Yes
function_kwargs_compressed	Annotated[Any, BytesAnnotation] | None	Yes
id	int	Yes
record_id	int	Yes
required_programs	list[str]	Yes

id: int

record_id: int

function: str | None

function_kwargs_compressed: QCPortalBytes | None

compute_tag: str

compute_priority: PriorityEnum

required_programs: list[str]

property tag: str

property priority: PriorityEnum

property function_kwargs: dict[str, Any] | None

class ServiceDependency

Bases: BaseModel

Fields

Field	Type	Required	Default
extras	dict[str, Any]	Yes
record_id	int	Yes

record_id: int

extras: dict[str, Any]

class RecordService

Bases: BaseModel

Fields

Field	Type	Required	Default
compute_priority	PriorityEnum	Yes
compute_tag	str	Yes
dependencies	list[ServiceDependency]	Yes
find_existing	bool	Yes
id	int	Yes
record_id	int	Yes
service_state	dict[str, Any] | None	No	None

id: int

record_id: int

compute_tag: str

compute_priority: PriorityEnum

find_existing: bool

service_state: dict[str, Any] | None

dependencies: list[ServiceDependency]

class BaseRecord

Bases: BaseModel

Fields

Field	Type	Required	Default	Alias
comments_	list[RecordComment] | None	No	None	comments
compute_history_	list[ComputeHistory] | None	No	None	compute_history
created_on	datetime	Yes
creator_user	str | None	Yes
description	str | None	No	None
extras	dict[str, Any]	No	{}
id	int	Yes
is_service	bool	Yes
manager_name	str | None	Yes
modified_on	datetime	Yes
name	str | None	No	None
native_files_	dict[str, NativeFile] | None	No	None	native_files
properties	dict[str, Any] | None	Yes
record_type	str	Yes
service_	RecordService | None	No	None	service
status	RecordStatusEnum	Yes
tags	list[str] | None	No	None
task_	RecordTask | None	No	None	task

id: int

record_type: str

is_service: bool

name: str | None

description: str | None

tags: list[str] | None

properties: dict[str, Any] | None

extras: dict[str, Any]

status: RecordStatusEnum

manager_name: str | None

created_on: datetime

modified_on: datetime

creator_user: str | None

compute_history_: list[ComputeHistory] | None

task_: RecordTask | None

service_: RecordService | None

comments_: list[RecordComment] | None

native_files_: dict[str, NativeFile] | None

__init__(client=None, base_url_prefix=None, **kwargs)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

base_url_prefix (str | None)

classmethod get_subclass(record_type)

Obtain a subclass of this class given its record_type

Parameters:

record_type (str)

Return type:

Type[BaseRecord]

classmethod fetch_children_multi(records, include=None, force_fetch=False)

Fetches all children of the given records

This tries to work efficiently, fetching larger batches of children that can span multiple records

Parameters:

• records (Iterable[BaseRecord | None])

• include (Iterable[str] | None)

• force_fetch (bool)

fetch_children(include=None, force_fetch=False)

Fetches all children of this record recursively

Parameters:

• include (Iterable[str] | None)

• force_fetch (bool)

sync_to_cache(detach=False)

Syncs this record to the cache

If detach is True, then the record will be removed from the cache

Parameters:

detach (bool)

get_cache_dict(**kwargs)

Returns a dictionary of the record meant for caching

When a record is stored in the cache, it is stored without child records (which are in the cache under their own keys). This function returns a dictionary that excludes those children.

kwargs are passed directly to the pydantic dict() function.

Return type:

dict[str, Any]

propagate_client(client, base_url_prefix)

Propagates a client and related information to this record to any fields within this record that need it

This is expected to be called from derived class propagate_client functions as well

Parameters:

base_url_prefix (str | None)

property offline: bool

property children_status: dict[RecordStatusEnum, int]

Returns a dictionary of the status of all children of this record

property children_errors: list[BaseRecord]

Returns errored child records

property compute_history: list[ComputeHistory]

property task: RecordTask | None

property service: RecordService | None

get_waiting_reason()

Return type:

dict[str, Any]

property comments: list[RecordComment] | None

property native_files: dict[str, NativeFile] | None

property stdout: str | None

property stderr: str | None

property error: dict[str, Any] | None

property provenance: Provenance | None

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

class RecordAddBodyBase

Bases: RestModelBase

Fields

Field	Type	Required	Default
compute_priority	PriorityEnum	Yes
compute_tag	str	Yes
find_existing	bool	No	True

compute_tag: LowerStr

compute_priority: PriorityEnum

find_existing: bool

class RecordModifyBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
comment	str | None	No	None
compute_priority	PriorityEnum | None	No	None
compute_tag	Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=None, max_length=None, pattern=None, ascii_only=None)] | None	No	None
record_ids	list[int]	Yes
status	RecordStatusEnum | None	No	None

record_ids: list[int]

status: RecordStatusEnum | None

compute_priority: PriorityEnum | None

compute_tag: LowerStr | None

comment: str | None

class RecordDeleteBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_children	bool	Yes
record_ids	list[int]	Yes
soft_delete	bool	Yes

record_ids: list[int]

soft_delete: bool

delete_children: bool

class RecordRevertBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
record_ids	list[int]	Yes
revert_status	RecordStatusEnum	Yes

revert_status: RecordStatusEnum

record_ids: list[int]

class RecordQueryFilters

Bases: QueryModelBase

Fields

Field	Type	Required	Default
child_id	list[int] | None	No	None
created_after	datetime | None	No	None
created_before	datetime | None	No	None
creator_user	list[int | str] | None	No	None
cursor	int | None	No	None
dataset_id	list[int] | None	No	None
history_manager_name	list[str] | None	No	None
limit	int | None	No	None
manager_name	list[str] | None	No	None
modified_after	datetime | None	No	None
modified_before	datetime | None	No	None
parent_id	list[int] | None	No	None
project_id	list[int] | None	No	None
record_id	list[int] | None	No	None
record_type	list[str] | None	No	None
status	list[RecordStatusEnum] | None	No	None

Validators

Validator	Mode	Fields
parse_dates	before	created_before, created_after, modified_before, modified_after
validate_lists	before	limit, cursor

record_id: list[int] | None

record_type: list[str] | None

manager_name: list[str] | None

history_manager_name: list[str] | None

status: list[RecordStatusEnum] | None

dataset_id: list[int] | None

project_id: list[int] | None

parent_id: list[int] | None

child_id: list[int] | None

created_before: datetime | None

created_after: datetime | None

modified_before: datetime | None

modified_after: datetime | None

creator_user: list[int | str] | None

classmethod parse_dates(v)

classmethod validate_lists(v)

limit

cursor

class RecordQueryIterator

Bases: QueryIteratorBase[_Record_T]

Iterator for all types of record queries

This iterator transparently handles batching and pagination over the results of a record query and works with all kinds of records.

__init__(client, query_filters, record_type, include=None)

Construct an iterator

Parameters:

• client – QCPortal client object used to contact/retrieve data from the server

• query_filters (RecordQueryFilters) – The actual query information to send to the server

• record_type (Type[_Record_T]) – What type of record we are querying for

• include (Iterable[str] | None)

reset()

Starts retrieval of results from the beginning again

record_from_dict(data, client=None, base_url_prefix=None)

Create a record object from a dictionary containing the record information

This determines the appropriate record class (deriving from BaseRecord) and creates an instance of that class.

Parameters:

• data (dict[str, Any])

• client (Any)

• base_url_prefix (str | None)

Return type:

BaseRecord

records_from_dicts(data, client=None, base_url_prefix=None)

Create a list of record objects from a sequence of datamodels

This determines the appropriate record class (deriving from BaseRecord) and creates an instance of that class.

Parameters:

• data (Sequence[dict[str, Any] | None])

• client (Any)

• base_url_prefix (str | None)

Return type:

list[BaseRecord | None]

compare_base_records(record_1, record_2)

Compares two QCPortal record objects

Parameters:

• record_1 (BaseRecord)

• record_2 (BaseRecord)

class DatasetAttachmentType

Bases: str, Enum

The type of attachment a file is for a dataset

other = 'other'

view = 'view'

__init__(*args, **kwds)

class DatasetAttachment

Bases: ExternalFile

Fields

Field	Type	Required	Default
attachment_type	DatasetAttachmentType	Yes
created_on	datetime	Yes
description	str | None	Yes
file_name	str	Yes
file_size	int	Yes
file_type	ExternalFileTypeEnum	Yes
id	int	Yes
provenance	dict[str, Any]	Yes
sha256sum	str	Yes
status	ExternalFileStatusEnum	Yes

attachment_type: DatasetAttachmentType

download(destination_path, overwrite=False)

Downloads an external file to the given path

The file size and checksum will be checked against the metadata stored on the server

Parameters:

• destination_path (str) – Full path to the destination file (including filename)

• overwrite (bool) – If True, allow for overwriting an existing file. If False, and a file already exists at the given destination path, an exception will be raised.

Return type:

None

get_direct_url()

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

propagate_client(client)

id

file_type

created_on

status

file_name

description

provenance

sha256sum

file_size

class Citation

Bases: BaseModel

A literature citation.

Fields

Field	Type	Required	Default
acs_citation	str | None	No	None
bibtex	str | None	No	None
doi	str | None	No	None
url	str | None	No	None

acs_citation: str | None

bibtex: str | None

doi: str | None

url: str | None

to_acs()

Returns an ACS-formatted citation

Return type:

str

class ContributedValues

Bases: BaseModel

Fields

Field	Type	Required	Default
citations	list[Citation] | None	No	None
comments	str | None	No	None
doi	str | None	No	None
external_url	str | None	No	None
index	Any	Yes
name	str	Yes
theory_level	str | dict[str, str]	Yes
theory_level_details	str | dict[str, str | None] | None	No	None
units	str	Yes
values	Any	Yes
values_structure	dict[str, Any]	No	{}

name: str

values: Any

index: Any

values_structure: dict[str, Any]

theory_level: str | dict[str, str]

units: str

theory_level_details: str | dict[str, str | None] | None

citations: list[Citation] | None

external_url: str | None

doi: str | None

comments: str | None

class BaseDataset

Bases: BaseModel

Fields

Field	Type	Required	Default	Alias
attachments_	list[DatasetAttachment] | None	No	None	attachments
auto_fetch_missing	bool	No	True
contributed_values_	dict[str, ContributedValues] | None	No	None	contributed_values
creator_user	str | None	Yes
dataset_type	str	Yes
default_compute_priority	PriorityEnum	Yes
default_compute_tag	str	Yes
description	str	Yes
extras	dict[str, Any]	Yes
id	int	Yes
name	str	Yes
provenance	dict[str, Any]	Yes
tagline	str	Yes
tags	list[str]	Yes

id: int

dataset_type: str

name: str

description: str

tagline: str

tags: list[str]

provenance: dict[str, Any]

extras: dict[str, Any]

default_compute_tag: str

default_compute_priority: PriorityEnum

creator_user: str | None

contributed_values_: dict[str, ContributedValues] | None

attachments_: list[DatasetAttachment] | None

auto_fetch_missing: bool

__init__(client=None, base_url_prefix=None, cache_data=None, **kwargs)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

• client (PortalClient | None)

• base_url_prefix (str | None)

• cache_data (DatasetCache | None)

classmethod get_subclass(dataset_type)

Parameters:

dataset_type (str)

propagate_client(client, base_url_prefix)

Propagates a client to this record to any fields within this record that need it

This may also be called from derived class propagate_client functions as well

Parameters:

base_url_prefix (str | None)

submit(entry_names=None, specification_names=None, compute_tag=None, compute_priority=None, find_existing=True, **kwargs)

Create records for this dataset

This function actually populates the datasets records given the entry and specification information.

Parameters:

• entry_names (str | Iterable[str] | None) – Submit only records for these entries

• specification_names (str | Iterable[str] | None) – Submit only records for these specifications

• compute_tag (str | None) – Use this compute tag for submissions (overrides the dataset default tag)

• compute_priority (PriorityEnum | None) – Use this compute priority for submissions (overrides the dataset default priority)

• find_existing (bool) – If True, the database will be searched for existing records that match the requested calculations, and new records created for those that don’t match. If False, new records will always be created.

Return type:

InsertCountsMetadata

background_submit(entry_names=None, specification_names=None, compute_tag=None, compute_priority=None, find_existing=True, **kwargs)

Adds a dataset submission internal job to the server

This internal job is the one to actually do the submission, which can take a while.

You can check the progress of the internal job using the return object.

See submit() for info on the function parameters.

Returns:

An internal job object that can be watch or used to determine the progress of the job.

Parameters:

• entry_names (str | Iterable[str] | None)

• specification_names (str | Iterable[str] | None)

• compute_tag (str | None)

• compute_priority (PriorityEnum)

• find_existing (bool)

Return type:

InternalJob

get_internal_job(job_id)

Parameters:

job_id (int)

Return type:

InternalJob

list_internal_jobs(status=None)

Parameters:

status (InternalJobStatusEnum | Iterable[InternalJobStatusEnum] | None)

Return type:

list[InternalJob]

fetch_attachments()

property attachments: list[DatasetAttachment]

delete_attachment(file_id)

Parameters:

file_id (int)

download_attachment(attachment_id, destination_path=None, overwrite=True)

Downloads an attachment

If destination path is not given, the file will be placed in the current directory, and the filename determined by what is stored on the server.

Parameters:

• attachment_id (int) – ID of the attachment to download. See the attachments property

• destination_path (str | None) – Full path to the destination file (including filename)

• overwrite (bool) – If True, any existing file will be overwritten

list_views()

download_view(view_file_id=None, destination_path=None, overwrite=True)

Downloads a view for this dataset

If a view_file_id is not given, the most recent view will be downloaded.

If destination path is not given, the file will be placed in the current directory, and the filename determined by what is stored on the server.

Parameters:

• view_file_id (int | None) – ID of the view to download. See list_views(). If None, will download the latest view

• destination_path (str | None) – Full path to the destination file (including filename)

• overwrite (bool) – If True, any existing file will be overwritten

use_view_cache(view_file_path)

Loads a vuew for this dataset as a cache file

Parameters:

view_file_path (str) – Full path to the view file

preload_cache(view_file_id=None)

Downloads a view file and uses it as the current cache

Parameters:

view_file_id (int | None) – ID of the view to download. See list_views(). If None, will download the latest view

create_view(description, provenance, status=None, include=None, exclude=None, *, include_children=True)

Creates a view of this dataset on the server

This function will return an InternalJob which can be used to watch for completion if desired. The job will run server side without user interaction.

Note the ID field of the object if you with to retrieve this internal job later (via list_internal_jobs() or PortalClient.get_internal_job)

Parameters:

• description (str) – String describing the view file

• provenance (dict[str, Any]) – Dictionary with any metadata or other information about the view. Information regarding the options used to create the view will be added.

• status (Iterable[RecordStatusEnum] | None) – List of statuses to include. Default is to include records with any status

• include (Iterable[str] | None) – List of specific record fields to include in the export. Default is to include most fields

• exclude (Iterable[str] | None) – List of specific record fields to exclude from the export. Defaults to excluding none.

• include_children (bool) – Specifies whether child records associated with the main records should also be included (recursively) in the view file.

Returns:

An InternalJob object which can be used to watch for completion.

Return type:

InternalJob

property is_view: bool

status()

Returns the status of the dataset’s computations, broken down by specification

The status is computed on the server, and does not require downloading any records.

Returns:

A dictionary with specification names as keys. Each value is itself a dictionary mapping record status to the number of records of the dataset with that status. Statuses with no records are not present.

Return type:

dict[str, Any]

status_table()

Returns the status of the dataset’s computations as a table (in a string)

Return type:

str

print_status()

Return type:

None

detailed_status()

Returns the status of every record of the dataset individually

Unlike status(), nothing is grouped or counted - there is one entry per record. This is what to use when you need to know which entries are in a particular state.

Returns:

A list of tuples (entry name, specification name, status), in no particular order

Return type:

list[tuple[str, str, RecordStatusEnum]]

status_by_compute_tag()

Returns the status of the dataset’s computations, broken down by compute tag

Only records that still have an entry in the task or service queue are counted. A record’s task is removed from the queue when it completes, so completed records do not appear here, and these counts will not sum to the counts returned by status(). In practice, this function reports the waiting, running, and errored records of the dataset - that is, the work that is still outstanding, and which compute tag it is queued under.

Returns:

A list of tuples (compute tag, status, number of records), in no particular order

Return type:

list[tuple[str, RecordStatusEnum, int]]

property offline: bool

assert_online()

property record_count: int

property computed_properties

assert_is_not_view()

set_name(new_name)

Parameters:

new_name (str)

set_description(new_description)

Parameters:

new_description (str)

set_tags(new_tags)

Parameters:

new_tags (list[str])

set_tagline(new_tagline)

Parameters:

new_tagline (str)

set_provenance(new_provenance)

Parameters:

new_provenance (dict[str, Any])

set_extras(new_extras)

Parameters:

new_extras (dict[str, Any])

set_default_compute_tag(new_default_compute_tag)

Parameters:

new_default_compute_tag (str)

set_default_compute_priority(new_default_compute_priority)

Parameters:

new_default_compute_priority (PriorityEnum)

property metadata: dict[str, Any]

set_metadata(new_metadata)

Parameters:

new_metadata (dict[str, Any])

property default_tag: str

set_default_tag(new_default_tag)

Parameters:

new_default_tag (str)

property default_priority: PriorityEnum

set_default_priority(new_default_priority)

Parameters:

new_default_priority (PriorityEnum)

property visibility: bool

set_visibility(new_visibility)

Parameters:

new_visibility (bool)

property group: str

set_group(new_group)

Parameters:

new_group (str)

fetch_specification_names()

Fetch all entry names from the remote server

These are fetched and then stored internally, and not returned.

Return type:

None

fetch_specifications(specification_names=None, force_refetch=False)

Fetch specifications from the remote server, storing them internally

Parameters:

• specification_names (str | Iterable[str] | None) – Names of specifications to fetch. If None, fetch all specifications

• force_refetch (bool) – If true, fetch data from the server even if it already exists locally

Return type:

None

property specification_names: list[str]

property specifications: Mapping[str, Any]

rename_specification(old_name, new_name)

Parameters:

• old_name (str)

• new_name (str)

delete_specification(name, delete_records=False)

Parameters:

• name (str)

• delete_records (bool)

Return type:

DeleteMetadata

fetch_entry_names()

Fetch all entry names from the remote server

These are fetched and then stored internally, and not returned.

Return type:

None

fetch_entries(entry_names=None, force_refetch=False)

Fetches entry information from the remote server, storing it internally

By default, already-fetched entries will not be fetched again, unless force_refetch is True.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of entries to fetch. If None, fetch all entries

• force_refetch (bool) – If true, fetch data from the server even if it already exists locally

Return type:

None

get_entry(entry_name, force_refetch=False)

Obtain entry information

The entry will be automatically fetched from the remote server if needed.

Parameters:

• entry_name (str)

• force_refetch (bool)

Return type:

Any | None

iterate_entries(entry_names=None, force_refetch=False)

Iterate over all entries

This is used as a generator, and automatically fetches entries as needed

Parameters:

• entry_names (str | Iterable[str] | None) – Names of entries to iterate over. If None, iterate over all entries

• force_refetch (bool) – If true, fetch data from the server even if it already exists locally

property entry_names: list[str]

rename_entries(name_map)

Renames entries in the dataset based on the provided mapping.

This method updates the names of entries both on the server and in the local cache. It ensures that the dataset is not a view and is online before proceeding with the renaming.

Parameters:

name_map (dict[str, str]) – A dictionary mapping old entry names to new entry names. Entries where the old name is the same as the new name are ignored.

Raises:

AssertionError – If the dataset is a view or is not online.

modify_entries(attribute_map=None, comment_map=None, overwrite_attributes=False)

Modifies the entries in the dataset by updating their attributes or comments.

Parameters:

• attribute_map (dict[str, dict[str, Any]] | None) – A dictionary mapping entry names to their updated attributes. Each entry name maps to a dictionary of attribute key-value pairs to be updated.

• comment_map (dict[str, str] | None) – A dictionary mapping entry names to their updated comments.

• overwrite_attributes (bool) – If True, existing attributes for the specified entries will be completely replaced by the provided attributes in attribute_map. If False, only the specified attributes will be updated, leaving others unchanged.

Raises:

AssertionError – If the dataset is a view or if the client is offline.

delete_entries(names, delete_records=False)

Deletes entries from the dataset.

Parameters:

• names (str | Iterable[str]) – The name or list of names of the entries to delete.

• delete_records (bool) – If True, associated records will also be deleted. Defaults to False.

Returns:

Metadata about the deletion operation.

Raises:

AssertionError – If the dataset is a view or not online.

Return type:

DeleteMetadata

fetch_records(entry_names=None, specification_names=None, status=None, include=None, fetch_updated=True, force_refetch=False)

Fetches record information from the remote server, storing it internally

By default, this function will only fetch records that have not been fetch previously. If force_refetch is True, then this will always fetch the records.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to fetch. If None, fetch all entries

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to fetch. If None, fetch all specifications

• status (RecordStatusEnum | Iterable[RecordStatusEnum] | None) – Fetch only records with these statuses

• include (Iterable[str] | None) – Additional fields to include in the returned record

• fetch_updated (bool) – Fetch any records that exist locally but have been updated on the server

• force_refetch (bool) – If true, fetch data from the server even if it already exists locally

get_record(entry_name, specification_name, include=None, fetch_updated=True, force_refetch=False)

Retrieve a calculation record associated with this dataset.

This method fetches the record from the remote server if it is not already cached locally. If the record does not exist for the specified entry and specification, it returns None.

Parameters:

• entry_name (str) – The name of the entry for which the record is to be retrieved.

• specification_name (str) – The name of the specification for which the record is to be retrieved.

• include (Iterable[str] | None) – Additional fields to include in the fetched record, by default None.

• fetch_updated (bool) – If True, fetches updated records from the server if they have been modified, by default True.

• force_refetch (bool) – If True, forces a refetch of the record from the server, ignoring the local cache, by default False.

Returns:

The calculation record associated with the specified entry and specification, or None if it does not exist.

Return type:

BaseRecord | None

iterate_records(entry_names=None, specification_names=None, status=None, include=None, fetch_updated=True, force_refetch=False)

Parameters:

• entry_names (str | Iterable[str] | None)

• specification_names (str | Iterable[str] | None)

• status (RecordStatusEnum | Iterable[RecordStatusEnum] | None)

• include (Iterable[str] | None)

• fetch_updated (bool)

• force_refetch (bool)

remove_records(entry_names, specification_names, delete_records=False)

Parameters:

• entry_names (str | Iterable[str])

• specification_names (str | Iterable[str])

• delete_records (bool)

Return type:

None

modify_records(entry_names=None, specification_names=None, new_compute_tag=None, new_compute_priority=None, new_comment=None, *, refetch_records=False, **kwargs)

Modify the compute tag, compute priority, or comment of records in this dataset.

Note: compute tags are not case sensitive and will be converted to lowercase.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to modify. If None, modify records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to modify. If None, modify records for all specifications.

• new_compute_tag (str | None) – The new compute tag to assign to the records.

• new_compute_priority (PriorityEnum | None) – The new compute priority to assign to the records.

• new_comment (str | None) – A new comment to add to the records.

• refetch_records (bool) – If True, refetch the modified records from the server.

reset_records(entry_names=None, specification_names=None, *, refetch_records=False)

Resets running or errored records to be waiting again.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to reset. If None, reset records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to reset. If None, reset records for all specifications.

• refetch_records (bool) – If True, refetch the reset records from the server.

cancel_records(entry_names=None, specification_names=None, *, refetch_records=False)

Marks running, waiting, or errored records as cancelled.

A cancelled record will not be picked up by a manager.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to cancel. If None, cancel records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to cancel. If None, cancel records for all specifications.

• refetch_records (bool) – If True, refetch the cancelled records from the server.

uncancel_records(entry_names=None, specification_names=None, *, refetch_records=False)

Undo the cancellation of records.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to uncancel. If None, uncancel records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to uncancel. If None, uncancel records for all specifications.

• refetch_records (bool) – If True, refetch the uncancelled records from the server.

invalidate_records(entry_names=None, specification_names=None, *, refetch_records=False)

Marks a completed record as invalid.

An invalid record is one that supposedly successfully completed. However, after review, is not correct.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to invalidate. If None, invalidate records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to invalidate. If None, invalidate records for all specifications.

• refetch_records (bool) – If True, refetch the invalidated records from the server.

uninvalidate_records(entry_names=None, specification_names=None, *, refetch_records=False)

Undo the invalidation of records.

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to uninvalidate. If None, uninvalidate records for all entries.

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to uninvalidate. If None, uninvalidate records for all specifications.

• refetch_records (bool) – If True, refetch the uninvalidated records from the server.

copy_entries_from(source_dataset_id, entry_names=None, existing_ok=False)

Copies entries from another dataset into this one

If entries already exist with the same name, an exception is raised.

Parameters:

• source_dataset_id (int) – The ID of the dataset to copy entries from

• entry_names (str | Iterable[str] | None) – Names of the entries to copy. If not provided, all entries will be copied.

• existing_ok (bool) – If False and an entry already exists by that name, an exception is raised.

copy_specifications_from(source_dataset_id, specification_names=None)

Copies specifications from another dataset into this one

If specifications already exist with the same name, an exception is raised.

Parameters:

• source_dataset_id (int) – The ID of the dataset to copy entries from

• specification_names (str | Iterable[str] | None) – Names of the specifications to copy. If not provided, all specifications will be copied.

copy_records_from(source_dataset_id, entry_names=None, specification_names=None, existing_ok=False)

Copies records from another dataset into this one

Entries and specifications will also be copied. If entries or specifications already exist with the same name, an exception is raised.

This does not actually fully copy records - the records will be linked to both datasets

Parameters:

• source_dataset_id (int) – The ID of the dataset to copy entries from

• entry_names (str | Iterable[str] | None) – Names of the entries to copy. If not provided, all entries will be copied.

• specification_names (str | Iterable[str] | None) – Names of the specifications to copy. If not provided, all specifications will be copied.

• existing_ok (bool) – If False and a specification or entry already exists by a supplied name, an exception is raised.

compile_values(value_call, value_names='value', entry_names=None, specification_names=None, unpack=False)

Compile values from records into a pandas DataFrame.

Parameters:

• value_call (Callable) – Function to call on each record to extract the desired value. Must return a scalar value or a sequence of values if ‘unpack’ is set to True.

• value_names (str | Sequence[str]) – Column name(s) for the extracted value(s). If a string is provided and multiple values are returned by ‘value_call’, columns are named by appending an index to this string. If a list of strings is provided, it must match the length of the sequence returned by ‘value_call’. Default is “value”.

• entry_names (str | Iterable[str] | None) – Entry names to filter records. If not provided, considers all entries.

• specification_names (str | Iterable[str] | None) – Specification names to filter records. If not provided, considers all specifications.

• unpack (bool) – If True, unpack the sequence of values returned by ‘value_call’ into separate columns. Default is False.

Returns:

A multi-index DataFrame where each row corresponds to an entry. Each column corresponds has a top level index as a specification, and a second level index as the appropriate value name. Values are extracted from records using ‘value_call’.

Return type:

pandas.DataFrame

Raises:

ValueError – If the length of ‘value_names’ does not match the number of values returned by ‘value_call’ when ‘unpack’ is set to True.

Notes

1. The DataFrame is structured such that the rows are entries and columns are specifications.

2. If ‘unpack’ is True, the function assumes ‘value_call’ returns a sequence of values that need to be distributed across columns in the resulting DataFrame. ‘value_call’ should always return the same number of values for each record if unpack is True.

get_properties_df(properties_list, entry_names=None, specification_names=None)

Retrieve a DataFrame populated with the specified properties from dataset records.

This function uses the provided list of property names to extract corresponding values from each record’s properties. It returns a DataFrame where rows represent each record. Each column corresponds has a top level index as a specification, and a second level index as the appropriate value name. Columns with all NaN values are dropped.

Parameters:

properties_list

List of property names to retrieve from the records.

entry_names

Entry names to filter records. If not provided, considers all entries.

specification_names

Specification names to filter records. If not provided, considers all specifications.

Returns:

pandas.DataFrame

A DataFrame populated with the specified properties for each record.

Parameters:

• properties_list (Sequence[str])

• entry_names (Sequence[str] | None)

• specification_names (Sequence[str] | None)

Return type:

DataFrame

refresh_cache(entry_names=None, specification_names=None)

Refreshes some information in the cache with information on the server

This can be used to fix some inconsistencies in the cache without deleting and starting over. For example, this can fix instances where the record attached to a given entry & specification has changed (new record id) due to renaming specifications and entries, or via remove_records followed by a submit without duplicate checking.

This will also fetch any updated records

Parameters:

• entry_names (str | Iterable[str] | None) – Names of the entries whose records to fetch. If None, fetch all entries

• specification_names (str | Iterable[str] | None) – Names of the specifications whose records to fetch. If None, fetch all specifications

fetch_contributed_values()

property contributed_values: dict[str, ContributedValues]

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

Return type:

None

class DatasetAddBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
default_compute_priority	PriorityEnum	Yes
default_compute_tag	str	Yes
description	str	Yes
existing_ok	bool	No	False
extras	dict[str, Any]	Yes
name	str	Yes
provenance	dict[str, Any]	Yes
tagline	str	Yes
tags	list[str]	Yes

name: str

description: str

tagline: str

tags: list[str]

provenance: dict[str, Any]

default_compute_tag: str

default_compute_priority: PriorityEnum

extras: dict[str, Any]

existing_ok: bool

class DatasetModifyMetadata

Bases: RestModelBase

Fields

Field	Type	Required	Default
default_compute_priority	PriorityEnum	Yes
default_compute_tag	str	Yes
description	str	Yes
extras	dict[str, Any] | None	Yes
name	str	Yes
provenance	dict[str, Any] | None	Yes
tagline	str	Yes
tags	list[str]	Yes

name: str

description: str

tags: list[str]

tagline: str

provenance: dict[str, Any] | None

extras: dict[str, Any] | None

default_compute_tag: str

default_compute_priority: PriorityEnum

class DatasetQueryModel

Bases: RestModelBase

Fields

Field	Type	Required	Default
dataset_name	str	Yes
dataset_type	str	Yes
exclude	list[str] | None	No	None
include	list[str] | None	No	None

dataset_type: str

dataset_name: str

include: list[str] | None

exclude: list[str] | None

class DatasetFetchSpecificationBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
missing_ok	bool	No	False
names	list[str]	Yes

names: list[str]

missing_ok: bool

class DatasetFetchEntryBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
missing_ok	bool	No	False
names	list[str]	Yes

names: list[str]

missing_ok: bool

class DatasetDeleteStrBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_records	bool	No	False
names	list[str]	Yes

names: list[str]

delete_records: bool

class DatasetRemoveRecordsBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_records	bool	No	False
entry_names	list[str]	Yes
specification_names	list[str]	Yes

entry_names: list[str]

specification_names: list[str]

delete_records: bool

class DatasetDeleteParams

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_records	bool	No	False

Validators

Validator	Mode	Fields
validate_lists	before	delete_records

delete_records: bool

classmethod validate_lists(v)

class DatasetCloneBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
new_dataset_name	str	Yes
source_dataset_id	int	Yes

source_dataset_id: int

new_dataset_name: str

class DatasetCopyFromBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
copy_entries	bool	No	False
copy_records	bool	No	False
copy_specifications	bool	No	False
entry_names	list[str] | None	No	None
source_dataset_id	int	Yes
specification_names	list[str] | None	No	None

source_dataset_id: int

entry_names: list[str] | None

specification_names: list[str] | None

copy_entries: bool

copy_specifications: bool

copy_records: bool

class DatasetFetchRecordsBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
entry_names	list[str]	Yes
specification_names	list[str]	Yes
status	list[RecordStatusEnum] | None	No	None

entry_names: list[str]

specification_names: list[str]

status: list[RecordStatusEnum] | None

class DatasetCreateViewBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
description	str	Yes
exclude	list[str] | None	No	(None,)
include	list[str] | None	No	(None,)
include_children	bool	No	(True,)
provenance	dict[str, Any]	Yes
status	list[RecordStatusEnum] | None	No	(None,)

description: str

provenance: dict[str, Any]

status: list[RecordStatusEnum] | None

include: list[str] | None

exclude: list[str] | None

include_children: bool

class DatasetSubmitBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
compute_priority	PriorityEnum | None	No	None
compute_tag	str | None	No	None
entry_names	list[str] | None	No	None
find_existing	bool	No	True
specification_names	list[str] | None	No	None

entry_names: list[str] | None

specification_names: list[str] | None

compute_tag: str | None

compute_priority: PriorityEnum | None

find_existing: bool

class DatasetRecordModifyBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
comment	str | None	No	None
compute_priority	PriorityEnum | None	No	None
compute_tag	Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=None, max_length=None, pattern=None, ascii_only=None)] | None	No	None
entry_names	list[str] | None	No	None
specification_names	list[str] | None	No	None
status	RecordStatusEnum | None	No	None

entry_names: list[str] | None

specification_names: list[str] | None

status: RecordStatusEnum | None

compute_priority: PriorityEnum | None

compute_tag: LowerStr | None

comment: str | None

class DatasetRecordRevertBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
entry_names	list[str] | None	No	None
revert_status	RecordStatusEnum	No	None
specification_names	list[str] | None	No	None

entry_names: list[str] | None

specification_names: list[str] | None

revert_status: RecordStatusEnum

class DatasetQueryRecords

Bases: RestModelBase

Fields

Field	Type	Required	Default
dataset_type	list[str] | None	No	None
record_id	list[int]	Yes

record_id: list[int]

dataset_type: list[str] | None

class DatasetDeleteEntryBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_records	bool	No	False
names	list[str]	Yes

names: list[str]

delete_records: bool

class DatasetDeleteSpecificationBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
delete_records	bool	No	False
names	list[str]	Yes

names: list[str]

delete_records: bool

class DatasetModifyEntryBody

Bases: RestModelBase

Fields

Field	Type	Required	Default
attribute_map	dict[str, dict[str, Any]] | None	No	None
comment_map	dict[str, str] | None	No	None
overwrite_attributes	bool	No	False

attribute_map: dict[str, dict[str, Any]] | None

comment_map: dict[str, str] | None

overwrite_attributes: bool

class DatasetGetInternalJobParams

Bases: RestModelBase

Fields

Field	Type	Required	Default
status	list[InternalJobStatusEnum] | None	No	None

status: list[InternalJobStatusEnum] | None

dataset_from_dict(data, client, base_url_prefix=None, cache_data=None)

Create a dataset object from a datamodel

This determines the appropriate dataset class (deriving from BaseDataset) and creates an instance of that class.

This works if the data is a datamodel object already or a dictionary

Parameters:

• data (dict[str, Any])

• client (Any)

• base_url_prefix (str | None)

• cache_data (DatasetCache | None)

Return type:

BaseDataset

load_dataset_view(file_path)

Parameters:

file_path (str)

Return type:

BaseDataset

dataset_from_cache(file_path)

Parameters:

file_path (str)

Return type:

BaseDataset

create_dataset_view(client, dataset_id, file_path, include=None, overwrite=False)

Parameters:

• client (PortalClient)

• dataset_id (int)

• file_path (str)

• include (Iterable[str] | None)

• overwrite (bool)