Base Record Models#

class PriorityEnum(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: int, Enum

The priority of a Task. Higher priority will be pulled first.

high = 2#
normal = 1#
low = 0#
class RecordStatusEnum(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The state of a record object. The states which are available are a finite set.

complete = 'complete'#
invalid = 'invalid'#
running = 'running'#
error = 'error'#
waiting = 'waiting'#
cancelled = 'cancelled'#
deleted = 'deleted'#
classmethod make_ordered_status(statuses)[source]#

Returns a list of the given statuses but in a defined order

Parameters:

statuses (Iterable[RecordStatusEnum])

Return type:

List[RecordStatusEnum]

class OutputTypeEnum(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

What type of data is stored

stdout = 'stdout'#
stderr = 'stderr'#
error = 'error'#
pydantic model OutputStore[source]#

Bases: BaseModel

Storage of outputs and error messages, with optional compression

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field output_type: OutputTypeEnum [Required]#

The type of output this is (stdout, error, etc)

field compression_type: CompressionEnum = CompressionEnum.none#

Compression method (such as lzma)

field data_: bytes | None = None (alias 'data')#
propagate_client(client, history_base_url)[source]#
property data: Any#
pydantic model ComputeHistory[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field id: int [Required]#
field record_id: int [Required]#
field status: RecordStatusEnum [Required]#
field manager_name: str | None = None#
field modified_on: datetime [Required]#
field provenance: Provenance | None = None#
field outputs_: Dict[str, OutputStore] | None = None (alias 'outputs')#
propagate_client(client, record_base_url)[source]#
fetch_all()[source]#
property outputs: Dict[str, OutputStore]#
get_output(output_type)[source]#
Parameters:

output_type (OutputTypeEnum)

Return type:

Any

property stdout: Any#
property stderr: Any#
property error: Any#
pydantic model NativeFile[source]#

Bases: BaseModel

Storage of native files, with compression

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field name: str [Required]#

Name of the file

field compression_type: CompressionEnum [Required]#

Compression method (such as lzma)

field data_: bytes | None = None (alias 'data')#
propagate_client(client, record_base_url)[source]#
fetch_all()[source]#
property data: Any#
save_file(directory, new_name=None, keep_compressed=False, overwrite=False)[source]#

Saves the file to the given directory

Parameters:

  • directory (str)

  • new_name (str | None)

  • keep_compressed (bool)

  • overwrite (bool)

pydantic model RecordInfoBackup[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field old_status: RecordStatusEnum [Required]#
field old_tag: str | None = None#
field old_priority: PriorityEnum | None = None#
field modified_on: datetime [Required]#
pydantic model RecordComment[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field id: int [Required]#
field record_id: int [Required]#
field username: str | None = None#
field timestamp: datetime [Required]#
field comment: str [Required]#
pydantic model RecordTask[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field id: int [Required]#
field record_id: int [Required]#
field function: str | None = None#
field function_kwargs_compressed: bytes | None = None#
field tag: str [Required]#
field priority: PriorityEnum [Required]#
field required_programs: List[str] [Required]#
property function_kwargs: Dict[str, Any] | None#
pydantic model ServiceDependency[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field record_id: int [Required]#
field extras: Dict[str, Any] [Required]#
pydantic model RecordService[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

Fields:
field id: int [Required]#
field record_id: int [Required]#
field tag: str [Required]#
field priority: PriorityEnum [Required]#
field find_existing: bool [Required]#
field service_state: Dict[str, Any] | None = None#
field dependencies: List[ServiceDependency] [Required]#
pydantic model BaseRecord[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • allow_mutation: bool = True

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
Validators:
field id: int [Required]#
field record_type: str [Required]#
field is_service: bool [Required]#
field properties: Dict[str, Any] | None = None#
field extras: Dict[str, Any] = {}#
Validated by:

  • _validate_extras

field status: RecordStatusEnum [Required]#
field manager_name: str | None = None#
field created_on: datetime [Required]#
field modified_on: datetime [Required]#
field owner_user: str | None = None#
field owner_group: str | None = None#
field compute_history_: List[ComputeHistory] | None = None (alias 'compute_history')#
field task_: RecordTask | None = None (alias 'task')#
field service_: RecordService | None = None (alias 'service')#
field comments_: List[RecordComment] | None = None (alias 'comments')#
field native_files_: Dict[str, NativeFile] | None = None (alias 'native_files')#
classmethod get_subclass(record_type)[source]#

Obtain a subclass of this class given its record_type

Parameters:

record_type (str)

Return type:

Type[BaseRecord]

classmethod fetch_children_multi(records, include=None, force_fetch=False)[source]#

Fetches all children of the given records

This tries to work efficiently, fetching larger batches of children that can span multiple records

Parameters:

  • records (Iterable[BaseRecord | None])

  • include (Iterable[str] | None)

  • force_fetch (bool)

fetch_children(include=None, force_fetch=False)[source]#

Fetches all children of this record recursively

Parameters:

  • include (Iterable[str] | None)

  • force_fetch (bool)

sync_to_cache(detach=False)[source]#

Syncs this record to the cache

If detach is True, then the record will be removed from the cache

Parameters:

detach (bool)

propagate_client(client)[source]#

Propagates a client and related information to this record to any fields within this record that need it

This is expected to be called from derived class propagate_client functions as well

property offline: bool#
property children_status: Dict[RecordStatusEnum, int]#

Returns a dictionary of the status of all children of this record

property children_errors: List[BaseRecord]#

Returns errored child records

property compute_history: List[ComputeHistory]#
property task: RecordTask | None#
property service: RecordService | None#
get_waiting_reason()[source]#
Return type:

Dict[str, Any]

property comments: List[RecordComment] | None#
property native_files: Dict[str, NativeFile] | None#
property stdout: str | None#
property stderr: str | None#
property error: Dict[str, Any] | None#
property provenance: Provenance | None#
pydantic model RecordAddBodyBase[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field tag: ConstrainedStrValue [Required]#
field priority: PriorityEnum [Required]#
field owner_group: str | None = None#
field find_existing: bool = True#
pydantic model RecordModifyBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field record_ids: List[int] [Required]#
field status: RecordStatusEnum | None = None#
field priority: PriorityEnum | None = None#
field tag: str | None = None#
field comment: str | None = None#
pydantic model RecordDeleteBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field record_ids: List[int] [Required]#
field soft_delete: bool [Required]#
field delete_children: bool [Required]#
pydantic model RecordRevertBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field revert_status: RecordStatusEnum [Required]#
field record_ids: List[int] [Required]#
pydantic model RecordQueryFilters[source]#

Bases: QueryModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
Validators:
field record_id: List[int] | None = None#
field record_type: List[str] | None = None#
field manager_name: List[str] | None = None#
field status: List[RecordStatusEnum] | None = None#
field dataset_id: List[int] | None = None#
field parent_id: List[int] | None = None#
field child_id: List[int] | None = None#
field created_before: datetime | None = None#
Validated by:
field created_after: datetime | None = None#
Validated by:
field modified_before: datetime | None = None#
Validated by:
field modified_after: datetime | None = None#
Validated by:
field owner_user: List[int | str] | None = None#
field owner_group: List[int | str] | None = None#
validator parse_dates  »  modified_before, modified_after, created_after, created_before[source]#
validator validate_lists  »  limit, cursor#
field limit: int | None = None#
Validated by:

  • validate_lists

field cursor: int | None = None#
Validated by:

  • validate_lists

class RecordQueryIterator(client, query_filters, record_type, include=None)[source]#

Bases: QueryIteratorBase[_Record_T]

Iterator for all types of record queries

This iterator transparently handles batching and pagination over the results of a record query, and works with all kinds of records.

Construct an iterator

Parameters:

  • client – QCPortal client object used to contact/retrieve data from the server

  • query_filters (RecordQueryFilters) – The actual query information to send to the server

  • record_type (Type[_Record_T]) – What type of record we are querying for

  • include (Optional[Iterable[str]])

reset()#

Starts retrieval of results from the beginning again

record_from_dict(data, client=None)[source]#

Create a record object from a dictionary containing the record information

This determines the appropriate record class (deriving from BaseRecord) and creates an instance of that class.

Parameters:

  • data (Dict[str, Any])

  • client (Any)

Return type:

BaseRecord

records_from_dicts(data, client=None)[source]#

Create a list of record objects from a sequence of datamodels

This determines the appropriate record class (deriving from BaseRecord) and creates an instance of that class.

Parameters:

  • data (Sequence[Dict[str, Any] | None])

  • client (Any)

Return type:

List[BaseRecord | None]

class DatasetAttachmentType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The type of attachment a file is for a dataset

other = 'other'#
view = 'view'#
pydantic model DatasetAttachment[source]#

Bases: ExternalFile

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:
field attachment_type: DatasetAttachmentType [Required]#
field id: int [Required]#
field file_type: ExternalFileTypeEnum [Required]#
field created_on: datetime [Required]#
field status: ExternalFileStatusEnum [Required]#
field file_name: str [Required]#
field description: str | None = None#
field provenance: Dict[str, Any] [Required]#
field sha256sum: str [Required]#
field file_size: int [Required]#
pydantic model Citation[source]#

Bases: BaseModel

A literature citation.

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • allow_mutation: bool = False

  • extra: Extra = Extra.forbid

Fields:
field acs_citation: str | None = None#
field bibtex: str | None = None#
field doi: str | None = None#
field url: str | None = None#
to_acs()[source]#

Returns an ACS-formatted citation

Return type:

str

pydantic model ContributedValues[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • allow_mutation: bool = False

  • extra: Extra = Extra.forbid

Fields:
field name: str [Required]#
field values: Any = None#
field index: Array [Required]#
Constraints:

  • type = array

  • items = {‘type’: ‘string’}

field values_structure: Dict[str, Any] = {}#
field theory_level: str | Dict[str, str] [Required]#
field units: str [Required]#
field theory_level_details: str | Dict[str, str | None] | None = None#
field citations: List[Citation] | None = None#
field external_url: str | None = None#
field doi: str | None = None#
field comments: str | None = None#
pydantic model BaseDataset[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • allow_mutation: bool = True

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field id: int [Required]#
field dataset_type: str [Required]#
field name: str [Required]#
field description: str [Required]#
field tagline: str [Required]#
field tags: List[str] [Required]#
field group: str [Required]#
field visibility: bool [Required]#
field provenance: Dict[str, Any] [Required]#
field default_tag: str [Required]#
field default_priority: PriorityEnum [Required]#
field owner_user: str | None = None#
field owner_group: str | None = None#
field metadata: Dict[str, Any] [Required]#
field extras: Dict[str, Any] [Required]#
field contributed_values_: Dict[str, ContributedValues] | None = None (alias 'contributed_values')#
field attachments_: List[DatasetAttachment] | None = None (alias 'attachments')#
field auto_fetch_missing: bool = True#
classmethod get_subclass(dataset_type)[source]#
Parameters:

dataset_type (str)

propagate_client(client)[source]#

Propagates a client to this record to any fields within this record that need it

This may also be called from derived class propagate_client functions as well

submit(entry_names=None, specification_names=None, tag=None, priority=None, find_existing=True)[source]#

Create records for this dataset

This function actually populates the datasets records given the entry and specification information.

Parameters:

  • entry_names (str | Iterable[str] | None) – Submit only records for these entries

  • specification_names (str | Iterable[str] | None) – Submit only records for these specifications

  • tag (str | None) – Use this tag for submissions (overrides the dataset default tag)

  • priority (PriorityEnum) – Use this tag for submissions (overrides the dataset default priority)

  • find_existing (bool) – If True, the database will be searched for existing records that match the requested calculations, and new records created for those that don’t match. If False, new records will always be created.

Return type:

InsertCountsMetadata

background_submit(entry_names=None, specification_names=None, tag=None, priority=None, find_existing=True)[source]#

Adds a dataset submission internal job to the server

This internal job is the one to actually do the submission, which can take a while.

You can check the progress of the internal job using the return object.

See submit() for info on the function parameters.

Returns:

An internal job object that can be watch or used to determine the progress of the job.

Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • tag (str | None)

  • priority (PriorityEnum)

  • find_existing (bool)

Return type:

InternalJob

get_internal_job(job_id)[source]#
Parameters:

job_id (int)

Return type:

InternalJob

list_internal_jobs(status=None)[source]#
Parameters:

status (InternalJobStatusEnum | Iterable[InternalJobStatusEnum] | None)

Return type:

List[InternalJob]

fetch_attachments()[source]#
property attachments: List[DatasetAttachment]#
delete_attachment(file_id)[source]#
Parameters:

file_id (int)

download_attachment(attachment_id, destination_path=None, overwrite=True)[source]#

Downloads an attachment

If destination path is not given, the file will be placed in the current directory, and the filename determined by what is stored on the server.

Parameters:

  • attachment_id (int) – ID of the attachment to download. See the attachments property

  • destination_path (str | None) – Full path to the destination file (including filename)

  • overwrite (bool) – If True, any existing file will be overwritten

list_views()[source]#
download_view(view_file_id=None, destination_path=None, overwrite=True)[source]#

Downloads a view for this dataset

If a view_file_id is not given, the most recent view will be downloaded.

If destination path is not given, the file will be placed in the current directory, and the filename determined by what is stored on the server.

Parameters:

  • view_file_id (int | None) – ID of the view to download. See list_views(). If None, will download the latest view

  • destination_path (str | None) – Full path to the destination file (including filename)

  • overwrite (bool) – If True, any existing file will be overwritten

use_view_cache(view_file_path)[source]#

Loads a vuew for this dataset as a cache file

Parameters:

view_file_path (str) – Full path to the view file

preload_cache(view_file_id=None)[source]#

Downloads a view file and uses it as the current cache

Parameters:

view_file_id (int | None) – ID of the view to download. See list_views(). If None, will download the latest view

create_view(description, provenance, status=None, include=None, exclude=None, *, include_children=True)[source]#

Creates a view of this dataset on the server

This function will return an InternalJob which can be used to watch for completion if desired. The job will run server side without user interaction.

Note the ID field of the object if you with to retrieve this internal job later (via get_internal_jobs() or PortalClient.get_internal_job)

Parameters:

  • description (str) – Optional string describing the view file

  • provenance (Dict[str, Any]) – Dictionary with any metadata or other information about the view. Information regarding the options used to create the view will be added.

  • status (Iterable[RecordStatusEnum] | None) – List of statuses to include. Default is to include records with any status

  • include (Iterable[str] | None) – List of specific record fields to include in the export. Default is to include most fields

  • exclude (Iterable[str] | None) – List of specific record fields to exclude from the export. Defaults to excluding none.

  • include_children (bool) – Specifies whether child records associated with the main records should also be included (recursively) in the view file.

Returns:

An InternalJob object which can be used to watch for completion.

Return type:

InternalJob

property is_view: bool#
status()[source]#
Return type:

Dict[str, Any]

status_table()[source]#

Returns the status of the dataset’s computations as a table (in a string)

Return type:

str

print_status()[source]#
Return type:

None

detailed_status()[source]#
Return type:

List[Tuple[str, str, RecordStatusEnum]]

property offline: bool#
assert_online()[source]#
property record_count: int#
property computed_properties#
assert_is_not_view()[source]#
set_name(new_name)[source]#
Parameters:

new_name (str)

set_description(new_description)[source]#
Parameters:

new_description (str)

set_visibility(new_visibility)[source]#
Parameters:

new_visibility (bool)

set_group(new_group)[source]#
Parameters:

new_group (str)

set_tags(new_tags)[source]#
Parameters:

new_tags (List[str])

set_tagline(new_tagline)[source]#
Parameters:

new_tagline (str)

set_provenance(new_provenance)[source]#
Parameters:

new_provenance (Dict[str, Any])

set_metadata(new_metadata)[source]#
Parameters:

new_metadata (Dict[str, Any])

set_default_tag(new_default_tag)[source]#
Parameters:

new_default_tag (str)

set_default_priority(new_default_priority)[source]#
Parameters:

new_default_priority (PriorityEnum)

fetch_specification_names()[source]#

Fetch all entry names from the remote server

These are fetched and then stored internally, and not returned.

Return type:

None

fetch_specifications(specification_names=None, force_refetch=False)[source]#

Fetch specifications from the remote server, storing them internally

Parameters:

  • specification_names (str | Iterable[str] | None) – Names of specifications to fetch. If None, fetch all specifications

  • force_refetch (bool) – If true, fetch data from the server even if it already exists locally

Return type:

None

property specification_names: List[str]#
property specifications: Mapping[str, Any]#
rename_specification(old_name, new_name)[source]#
Parameters:

  • old_name (str)

  • new_name (str)

delete_specification(name, delete_records=False)[source]#
Parameters:

  • name (str)

  • delete_records (bool)

Return type:

DeleteMetadata

fetch_entry_names()[source]#

Fetch all entry names from the remote server

These are fetched and then stored internally, and not returned.

Return type:

None

fetch_entries(entry_names=None, force_refetch=False)[source]#

Fetches entry information from the remote server, storing it internally

By default, already-fetched entries will not be fetched again, unless force_refetch is True.

Parameters:

  • entry_names (str | Iterable[str] | None) – Names of entries to fetch. If None, fetch all entries

  • force_refetch (bool) – If true, fetch data from the server even if it already exists locally

Return type:

None

get_entry(entry_name, force_refetch=False)[source]#

Obtain entry information

The entry will be automatically fetched from the remote server if needed.

Parameters:

  • entry_name (str)

  • force_refetch (bool)

Return type:

Any | None

iterate_entries(entry_names=None, force_refetch=False)[source]#

Iterate over all entries

This is used as a generator, and automatically fetches entries as needed

Parameters:

  • entry_names (str | Iterable[str] | None) – Names of entries to iterate over. If None, iterate over all entries

  • force_refetch (bool) – If true, fetch data from the server even if it already exists locally

property entry_names: List[str]#
rename_entries(name_map)[source]#
Parameters:

name_map (Dict[str, str])

modify_entries(attribute_map=None, comment_map=None, overwrite_attributes=False)[source]#
Parameters:

  • attribute_map (Dict[str, Dict[str, Any]] | None)

  • comment_map (Dict[str, str] | None)

  • overwrite_attributes (bool)

delete_entries(names, delete_records=False)[source]#
Parameters:

  • names (str | Iterable[str])

  • delete_records (bool)

Return type:

DeleteMetadata

fetch_records(entry_names=None, specification_names=None, status=None, include=None, fetch_updated=True, force_refetch=False)[source]#

Fetches record information from the remote server, storing it internally

By default, this function will only fetch records that have not been fetch previously. If force_refetch is True, then this will always fetch the records.

Parameters:

  • entry_names (str | Iterable[str] | None) – Names of the entries whose records to fetch. If None, fetch all entries

  • specification_names (str | Iterable[str] | None) – Names of the specifications whose records to fetch. If None, fetch all specifications

  • status (RecordStatusEnum | Iterable[RecordStatusEnum] | None) – Fetch only records with these statuses

  • include (Iterable[str] | None) – Additional fields to include in the returned record

  • fetch_updated (bool) – Fetch any records that exist locally but have been updated on the server

  • force_refetch (bool) – If true, fetch data from the server even if it already exists locally

get_record(entry_name, specification_name, include=None, fetch_updated=True, force_refetch=False)[source]#

Obtain a calculation record related to this dataset

The record will be automatically fetched from the remote server if needed. If a record does not exist for this entry and specification, None is returned

Parameters:

  • entry_name (str)

  • specification_name (str)

  • include (Iterable[str] | None)

  • fetch_updated (bool)

  • force_refetch (bool)

Return type:

BaseRecord | None

iterate_records(entry_names=None, specification_names=None, status=None, include=None, fetch_updated=True, force_refetch=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • status (RecordStatusEnum | Iterable[RecordStatusEnum] | None)

  • include (Iterable[str] | None)

  • fetch_updated (bool)

  • force_refetch (bool)

remove_records(entry_names=None, specification_names=None, delete_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • delete_records (bool)

Return type:

DeleteMetadata

modify_records(entry_names=None, specification_names=None, new_tag=None, new_priority=None, new_comment=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • new_tag (str | None)

  • new_priority (PriorityEnum | None)

  • new_comment (str | None)

  • refetch_records (bool)

reset_records(entry_names=None, specification_names=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • refetch_records (bool)

cancel_records(entry_names=None, specification_names=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • refetch_records (bool)

uncancel_records(entry_names=None, specification_names=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • refetch_records (bool)

invalidate_records(entry_names=None, specification_names=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • refetch_records (bool)

uninvalidate_records(entry_names=None, specification_names=None, *, refetch_records=False)[source]#
Parameters:

  • entry_names (str | Iterable[str] | None)

  • specification_names (str | Iterable[str] | None)

  • refetch_records (bool)

copy_entries_from(source_dataset_id, entry_names=None)[source]#

Copies entries from another dataset into this one

If entries already exist with the same name, an exception is raised.

Parameters:

  • source_dataset_id (int) – The ID of the dataset to copy entries from

  • entry_names (str | Iterable[str] | None) – Names of the entries to copy. If not provided, all entries will be copied.

copy_specifications_from(source_dataset_id, specification_names=None)[source]#

Copies specifications from another dataset into this one

If specifications already exist with the same name, an exception is raised.

Parameters:

  • source_dataset_id (int) – The ID of the dataset to copy entries from

  • specification_names (str | Iterable[str] | None) – Names of the specifications to copy. If not provided, all specifications will be copied.

copy_records_from(source_dataset_id, entry_names=None, specification_names=None)[source]#

Copies records from another dataset into this one

Entries and specifications will also be copied. If entries or specifications already exist with the same name, an exception is raised.

This does not actually fully copy records - the records will be linked to both datasets

Parameters:

  • source_dataset_id (int) – The ID of the dataset to copy entries from

  • entry_names (str | Iterable[str] | None) – Names of the entries to copy. If not provided, all entries will be copied.

  • specification_names (str | Iterable[str] | None) – Names of the specifications to copy. If not provided, all specifications will be copied.

compile_values(value_call, value_names='value', entry_names=None, specification_names=None, unpack=False)[source]#

Compile values from records into a pandas DataFrame.

Parameters:

  • value_call (Callable) – Function to call on each record to extract the desired value. Must return a scalar value or a sequence of values if ‘unpack’ is set to True.

  • value_names (Union[Sequence[str], str]) – Column name(s) for the extracted value(s). If a string is provided and multiple values are returned by ‘value_call’, columns are named by appending an index to this string. If a list of strings is provided, it must match the length of the sequence returned by ‘value_call’. Default is “value”.

  • entry_names (Optional[Union[str, Iterable[str]]]) – Entry names to filter records. If not provided, considers all entries.

  • specification_names (Optional[Union[str, Iterable[str]]]) – Specification names to filter records. If not provided, considers all specifications.

  • unpack (bool) – If True, unpack the sequence of values returned by ‘value_call’ into separate columns. Default is False.

Returns:

A multi-index DataFrame where each row corresponds to an entry. Each column corresponds has a top level index as a specification, and a second level index as the appropriate value name. Values are extracted from records using ‘value_call’.

Return type:

pandas.DataFrame

Raises:

ValueError – If the length of ‘value_names’ does not match the number of values returned by ‘value_call’ when ‘unpack’ is set to True.

Notes

  1. The DataFrame is structured such that the rows are entries and columns are specifications.

2. If ‘unpack’ is True, the function assumes ‘value_call’ returns a sequence of values that need to be distributed across columns in the resulting DataFrame. ‘value_call’ should always return the same number of values for each record if unpack is True.

get_properties_df(properties_list)[source]#

Retrieve a DataFrame populated with the specified properties from dataset records.

This function uses the provided list of property names to extract corresponding values from each record’s properties. It returns a DataFrame where rows represent each record. Each column corresponds has a top level index as a specification, and a second level index as the appropriate value name. Columns with all NaN values are dropped.

Parameters:#

properties_list

List of property names to retrieve from the records.

Returns:#

pandas.DataFrame

A DataFrame populated with the specified properties for each record.

Parameters:

properties_list (Sequence[str])

Return type:

DataFrame

refresh_cache(entry_names=None, specification_names=None)[source]#

Refreshes some information in the cache with information on the server

This can be used to fix some inconsistencies in the cache without deleting and starting over. For example, this can fix instances where the record attached to a given entry & specification has changed (new record id) due to renaming specifications and entries, or via remove_records followed by a submit without duplicate checking.

This will also fetch any updated records

Parameters:

  • entry_names (str | Iterable[str] | None) – Names of the entries whose records to fetch. If None, fetch all entries

  • specification_names (str | Iterable[str] | None) – Names of the specifications whose records to fetch. If None, fetch all specifications

fetch_contributed_values()[source]#
property contributed_values: Dict[str, ContributedValues]#
pydantic model DatasetAddBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field name: str [Required]#
field description: str [Required]#
field tagline: str [Required]#
field tags: List[str] [Required]#
field group: str [Required]#
field provenance: Dict[str, Any] [Required]#
field visibility: bool [Required]#
field default_tag: str [Required]#
field default_priority: PriorityEnum [Required]#
field metadata: Dict[str, Any] [Required]#
field owner_group: str | None = None#
field existing_ok: bool = False#
pydantic model DatasetModifyMetadata[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field name: str [Required]#
field description: str [Required]#
field tags: List[str] [Required]#
field tagline: str [Required]#
field group: str [Required]#
field visibility: bool [Required]#
field provenance: Dict[str, Any] | None = None#
field metadata: Dict[str, Any] | None = None#
field default_tag: str [Required]#
field default_priority: PriorityEnum [Required]#
pydantic model DatasetQueryModel[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field dataset_type: str | None = None#
field dataset_name: str | None = None#
field include: List[str] | None = None#
field exclude: List[str] | None = None#
pydantic model DatasetFetchSpecificationBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field names: List[str] [Required]#
field missing_ok: bool = False#
pydantic model DatasetFetchEntryBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field names: List[str] [Required]#
field missing_ok: bool = False#
pydantic model DatasetDeleteStrBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field names: List[str] [Required]#
field delete_records: bool = False#
pydantic model DatasetRemoveRecordsBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field entry_names: List[str] [Required]#
field specification_names: List[str] [Required]#
field delete_records: bool = False#
pydantic model DatasetDeleteParams[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
Validators:
field delete_records: bool = False#
Validated by:
validator validate_lists  »  delete_records[source]#
pydantic model DatasetCloneBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field source_dataset_id: int [Required]#
field new_dataset_name: str [Required]#
pydantic model DatasetCopyFromBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field source_dataset_id: int [Required]#
field entry_names: List[str] | None = None#
field specification_names: List[str] | None = None#
field copy_entries: bool = False#
field copy_specifications: bool = False#
field copy_records: bool = False#
pydantic model DatasetFetchRecordsBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field entry_names: List[str] [Required]#
field specification_names: List[str] [Required]#
field status: List[RecordStatusEnum] | None = None#
pydantic model DatasetCreateViewBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field description: str | None = None#
field provenance: Dict[str, Any] [Required]#
field status: List[RecordStatusEnum] | None = (None,)#
field include: List[str] | None = (None,)#
field exclude: List[str] | None = (None,)#
field include_children: bool = (True,)#
pydantic model DatasetSubmitBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field entry_names: List[str] | None = None#
field specification_names: List[str] | None = None#
field tag: str | None = None#
field priority: PriorityEnum | None = None#
field owner_group: str | None = None#
field find_existing: bool = True#
pydantic model DatasetRecordModifyBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field entry_names: List[str] | None = None#
field specification_names: List[str] | None = None#
field status: RecordStatusEnum | None = None#
field priority: PriorityEnum | None = None#
field tag: str | None = None#
field comment: str | None = None#
pydantic model DatasetRecordRevertBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field entry_names: List[str] | None = None#
field specification_names: List[str] | None = None#
field revert_status: RecordStatusEnum = None#
pydantic model DatasetQueryRecords[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field record_id: List[int] [Required]#
field dataset_type: List[str] | None = None#
pydantic model DatasetDeleteEntryBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field names: List[str] [Required]#
field delete_records: bool = False#
pydantic model DatasetDeleteSpecificationBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field names: List[str] [Required]#
field delete_records: bool = False#
pydantic model DatasetModifyEntryBody[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field attribute_map: Dict[str, Dict[str, Any]] | None = None#
field comment_map: Dict[str, str] | None = None#
field overwrite_attributes: bool = False#
pydantic model DatasetGetInternalJobParams[source]#

Bases: RestModelBase

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Config:

  • extra: Extra = Extra.forbid

  • validate_assignment: bool = True

Fields:
field status: List[InternalJobStatusEnum] | None = None#
dataset_from_dict(data, client, cache_data=None)[source]#

Create a dataset object from a datamodel

This determines the appropriate dataset class (deriving from BaseDataset) and creates an instance of that class.

This works if the data is a datamodel object already or a dictionary

Parameters:

  • data (Dict[str, Any])

  • client (Any)

  • cache_data (DatasetCache | None)

Return type:

BaseDataset

load_dataset_view(file_path)[source]#
Parameters:

file_path (str)

Return type:

BaseDataset

dataset_from_cache(file_path)[source]#
Parameters:

file_path (str)

Return type:

BaseDataset

create_dataset_view(client, dataset_id, file_path, include=None, overwrite=False)[source]#
Parameters:

  • client (PortalClient)

  • dataset_id (int)

  • file_path (str)

  • include (Optional[Iterable[str]])

  • overwrite (bool)

On this page

This Page