0.12.0 / 2019-11-06¶
The ability to handle very large datasets (1M+ entries) quickly and efficiently.
Store and compute Wavefunction information.
Build, serve, and export views for Datasets that can stored in journal supplementary information or services like Zenodo.
A new GUI dashboard to observe the current state of the server, see statistics, and fix issues.
(GH#433 and GH#462)
interface.collections) now have a
download`method which downloads a frozen view of the dataset. This view is used to speed up calls to
(GH#440) Wavefunctions can now be stored in the database using Result
(GH#453) The server now periodically logs manager and current state to provide data over time.
(GH#460) Contributed values are now in their own table to speed up access of Collections.
(GH#461) Services now update their corresponding record every iteration. An example is a torsiondrive which now updates the
optimization_historyfield each iteration.
(GH#429) Enables protocols for
(GH#430) Adds additional QCPortal type hints.
interface.collections) are now faster for calls to calls to
list_valuesfor large datasets if the server is configured to use frozen views. See “Server-side Dataset Views” documentation. Subsets may be passed to
(GH#447) Enables the creation of plaintext (xyz and csv) output from Dataset Collections.
(GH#455) Projection queries should now be much faster as excluded results are not pulled to the server.
(GH#458) Collections now have a metadata field.
FractalClient.list_collectionsby default only returns collections whose visibility flag is set to true, and whose group is “default”. This change was made to filter out in-progress, intermediate, and specialized collections.
(GH#464) Molecule insert speeds are now 4-16x faster.
interface.collections), the previously deprecated functions
list_historyhave been removed.
0.11.0 / 2019-10-01¶
(GH#420) Pre-storage data handling through Elemental’s
Protocolsfeature are now present in Fractal. Although only optimization protocols are implemented functionally, the database side has been upgraded to store protocol settings.
(GH#385, GH#404, GH#411)
ReactionDatasethave five new functions for accessing data.
get_valuesreturns the canonical headline value for a dataset (e.g. the interaction energy for S22) in data columns with caching, both for result-backed values and contributed values. This function replaces the now-deprecated
list_valuesreturns the list of data columns available from
get_values. This function replaces the now-deprecated
ResultRecordor a projection. For the case of
ReactionDataset, the results are broken down into component calculations. The function replaces the now-deprecated
list_recordsreturns the list of data columns available from
Moleculeassociated with a dataset.
(GH#393) A new feature added to
Clientto be able to have more custom and fast queries, the
custom_querymethod. Those fast queries are now used in
torsiondrive.get_final_results. More Advanced queries will be added.
managerselector fields to
client.query_tasks. This is helpful for managing jobs in the queue and detecting failures.
Datasetcollection’s primary indices (database level) have been updated to reflect its new understanding.
(GH#396) Fixed a bug in internal
Datasetfunction which caused
ComputeResponseto be truncated when the number of calculations is larger than the query_limit.
Dataset.get_valuesfor any method which involved DFTD3.
(GH#409) Fixed a compatibility bug in specific version of Intel-OpenMP by skipping version 2019.5-281.
0.10.0 / 2019-08-26¶
Stable Beta Release
This release marks Fractal’s official Stable Beta Release. This means that future, non-backwards compatible changes to the API will result in depreciation warnings.
(GH#356) Collections’ database representations have been improved to better support future upgrade paths.
(GH#375) Dataset Records are now copied alongside the Collections.
testingsuite from Fractal now exposes as a PyTest entry-point when Fractal is installed so that tests can be run from anywhere with the
--pyargs qcfractalflag of
(GH#384) “Dataset Records” and “Reaction Dataset Records” have been renamed to “Dataset Entry” and “Reaction Dataset Entry” respectively.
0.9.0 / 2019-08-16¶
(GH#354) Fractal now takes advantage of Elemental’s new Msgpack serialization option for Models. Serialization defaults to msgpack when available (
conda install msgpack-python [-c conda-forge]), falling back to JSON otherwise. This results in substantial speedups for both serialization and deserialization actions and should be a transparent replacement for users within Fractal, Engine, and Elemental themselves.
(GH#358) Fractal Server’s CLI now supports user manipulations through the
qcfractal-server usersubcommand. This allows server administrators to control users and their access without directly interacting with the storage socket.
get_historyfunction is fixed by allowing the ability to force a new query even if one has already been cached.
(GH#338) The Queue Manager which generated a
Resultis now stored in the
(GH#341) Skeletal Queue Manager YAML files can now be generated through the
--skeletonCLI flag on
(GH#361) Staged DB’s in Fractal copy Alembic alongside them.
(GH#363) A new REST API hook for services has been added so Clients can manage Services.
0.8.0 / 2019-07-25¶
PostgreSQL is now the only supported database backend.
Fractal has officially dropped support for MongoDB in favor of PostgreSQL as our database backend. Although MongoDB served the start of Fractal well, our database design as evolved since then and will be better served by PostgreSQL.
(GH#307, GH#319 GH#321) Fractal’s Server CLI has been overhauled to more intuitively and intelligently control Server creation, startup, configuration, and upgrade paths. This is mainly reflected in a Fractal Server config file, a config folder (default location
~/.qca, and sub-commands
qcfractal-server (command)CLI. See the full documentation for details
(GH#323) First implementation of the
GridOptimizationDatasetfor collecting Grid Optimization calculations. Not yet fully featured, but operational for users to start working with.
(GH#291) Tests have been formally added for the Queue Manager to reduce bugs in the future. They cannot test on actual Schedulers yet, but its a step in the right direction.
(GH#295) Quality of life improvement for Mangers which by default will be less noisy about heartbeats and trigger a heartbeat less frequently. Both options can still be controlled through verbosity and a config setting.
(GH#296) Services are now prioritized by the date they are created to properly order the compute queue.
TorsionDriveDatasetstatus can now be checked through the
.status()method which shows the current progress of the computed data.
(GH#310) The Client can now modify tasks and restart them if need be in the event of random failures.
(GH#313) Queue Managers now have more detailed statistics about failure rates, and core-hours consumed (estimated)
PostgresHarnesshas been improved to include better error handling if Postgress is not found, and will not try to stop/start if the target data directory is already configured and running.
Recordmodels and their derivatives now have a
get_molecule()method for fetching the molecule directly.
(GH#324) Optimization queries for its trajectory pull the entire trajectory in one go and keep the correct order.
get_trajectoryalso pulls the correct order.
(GH#325) Collections’ have been improved to be more efficient. Previous queries are cached locally and the
computecall is now a single function, removing the need to make a separate call to the submission formation.
ReactionDatasetnow explicitly groups the fragments to future-proof this method from upstream changes to
(GH#329) All API requests are now logged server side anonymously.
(GH#331) Queue Manager jobs can now auto-retry failed jobs a finite number of times through QCEngine’s retry capabilities. This will only catch RandomErrors and all other errors are raised normally.
(GH#332) SQLAlchemy layer on the PostgreSQL database has received significant polish
(GH#291) Queue Manager documentation generation works on Pydantic 0.28+. A number as-of-yet uncaught/unseen bugs were revealed in tests and have been fixed as well.
(GH#300) Errors thrown in the level between Managers and their Adapters now correctly return a
dictto be consistent with all other errors and not crash the Manager.
(GH#301) Invalid passwords present a helpful error message now instead of raising an Internal Server Error to the user.
(GH#306) The Manager CLI option
tasks-per-workeris correctly hyphens instead of underscores to be consistent with all other flags.
(GH#316) Queue Manager workarounds for older versions of Dask-Jobqueue and Parsl have been removed and implicit dependency on the newer versions of those Adapters is enforced on CLI usage of
qcfractal-manager. These packages are not required for Fractal, so their versions are only checked when specifically used in the Managers.
TorsionDriveDatasetwill no longer cause a failure in adding them to the database while still preserving de-duplication.
(GH#327) Jupyter Notebook syntax highlighting has been fixed on Fractal’s documentation pages.
(GH#331) The BaseModel/Settings auto-documentation function can no longer throw an error which prevents using the code.
0.7.2 / 2019-05-31¶
(GH#275) Documentation has been further updated to be more contiguous between pages.
(GH#276) Imports and type hints in Database objects have been improved to remove ambiguity and make imports easier to follow.
(GH#280) Optimizations queried in the database are done with a more efficient lazy
selectin. This should make queries much faster.
(GH#281) Database Migration tech has been moved to their own folder to keep them isolated from normal production code. This PR also called the testing database
test_qcarchivedbto avoid clashes with production DBs. Finally, a new keyword for testing geometry optimizations has been added.
0.7.1 / 2019-05-28¶
0.7.0 / 2019-05-27¶
Final MongoDB Supported Release
This is the last major release which support MongoDB. Fractal is moving towards a PostgreSQL for database to make upgrades more stable and because it is more suited to the nature of QCArchive Data. The upgrade path from MongoDB to PostgreSQL will be provided by the Fractal developers in the next release. Due to the complex nature of the upgrade, the PostgreSQL upgrade will through scripts which will be provided. After the PostgreSQL upgrade, there will be built-in utilities to upgrade the Database.
(GH#242) Parsl can now be used as an
Adapterin the Queue Managers.
(GH#247) The new
OptimizationDatasetcollection has been added! This collection returns a set of optimized molecular structures given an initial input.
(GH#254) The QCFractal Server Dashboard is now available through a Dash interface. Although not fully featured yet, future updates will improve this as features are requested.
(GH#260) Its now even easier to install Fractal/Portal through conda with pre-built environments on the
qcarchiveconda channel. This channel only provides environment files, no packages (and there are not plans to do so.)
(GH#269) The Fractal Snowflake project has been extended to work in Jupyter Notebooks. A Fractal Snowflake can be created with the
FractalSnowflakeHandlerinside of a Jupyter Session.
Database Compatibility Updates¶
(GH#256) API calls to Elemental 0.4 have been updated. This changes the hashing system and so upgrading your Fractal Server instance to this (or higher) will require an upgrade path to the indices.
GridOptimizationRecordsupports the helper function
get_final_moleculeswhich returns the set of molecules at each final, optimized grid point.
TorsionDriveRecordsupport the helper function
get_final_results, which is like
get_final_molecules, but for x
(GH#241) The visualization suite with Plotly has been made more general so it can be invoked in different classes. This particular PR updates the TorsionDriveDataSet objects.
(GH#243) TorsionDrives in Fractal now support the updated Torsion Drive API from the underlying package. This includes both the new arguments and the “extra constraints” features.
(GH#244) Tasks which fail are now more verbose in the log as to why they failed. This is additional information on top of the number of pass/fail.
(GH#246) Queue Manager
verbositylevel is now passed down into the adapter programs as well and the log file (if set) will continue to print to the terminal as well as the physical file.
(GH#247) Procedure classes now all derive from a common base class to be more consistent with one another and for any new Procedures going forward.
(GH#248) Jobs which fail, or cannot be returned correctly, from Queue Managers are now better handled in the Manager and don’t sit in the Manager’s internal buffer. They will attempt to be returned to the Server on later updates. If too many jobs become stale, the Manager will shut itself down for safety.
(GH#258 and GH#268) Fractal Queue Managers are now fully documented, both from the CLI and through the doc pages themselves. There have also been a few variables renamed and moved to be more clear the nature of what they do. See the PR for the renamed variables.
(GH#251) The Fractal Server now reports valid minimum/maximum allowed client versions. The Portal Client will try check these numbers against itself and fail to connect if it is not within the Server’s allowed ranges. Clients started from Fractal’s
interfacedo not make this check.
(GH#248) Fixed a bug in Queue Managers where the extra worker startup commands for the Dask Adapter were not being parsed correctly.
(GH#250) Record objects now correctly set their provenance time on object creation, not module import.
(GH#253) A spelling bug was fixed in GridOptimization which caused hashing to not be processed correctly.
(GH#270) LSF clusters not in
MBfor the units on memory by config are now auto-detected (or manually set) without large workarounds in the YAML file and the CLI file itself. Supports documented settings of LSF 9.1.3.
0.6.0 / 2019-03-30¶
0.5.5 / 2019-03-26¶
(GH#228) ReactionDatasets visualization statistics plots can now be generated through Plotly! This feature includes bar plots and violin plots and is designed for interactive use through websites, Jupyter notebooks, and more.
(GH#233) TorsionDrive Datasets have custom visualization statistics through Plotly! This allows plotting 1-D torsion scans against other ones.
(GH#226) LSF can now be specified for the Queue Managers for Dask Managers.
(GH#228) Plotly is an optional dependency overall, it is not required to run QCFractal or QCPortal but will be downloaded in some situations. If you don’t have Plotly installed, more graceful errors beyond just raw
(GH#234) Queue Managers now report the number of passed and failed jobs they return to the server and can also have verbose (debug level) outputs to the log.
(GH#234) Dask-driven Queue Managers can now be set to simply scale up to a fixed number of workers instead of trying to adapt the number of workers on the fly.
(GH#227) SGE Clusters specified in Queue Manager under Dask correctly process
job_extrafor additional scheduler headers. This is implemented in a stable way such that if the upstream Dask Jobqueue implements a fix, the Manager will keep working without needing to get a new release.
(GH#234) Fireworks managers now return the same pydantic models as every other manager instead of raw dictionaries.
0.5.4 / 2019-03-21¶
(GH#216) Jobs submitted to the queue can now be assigned a priority to be served out to the Managers.
(GH#219) Temporary, pop-up, local instances of
FractalServercan now be created through the
FractalSnowflake. This creates an instance of
FractalServer, with its database structure, which is entirely held in temporary storage and memory, all of which is deleted upon exit/stop. This feature is designed for those who want to tinker with Fractal without needed to create their own database or connect to a production
(GH#220) Queue Managers can now set the
scratch_directoryvariable that is passed to QCEngine and its workers.
(GH#216) Queue Managers now report what programs and procedures they have access to and will only pull jobs they think they can execute.
(GH#222) All of
FractalClient’s methods now have full docstrings and type annotations for clairy
(GH#222) Massive overhaul to the REST interface to simplify internal calls from the client and server side.
TorsionDriveDatasetobjects are modeled through pydantic objects to allow easier interface with the database back end and data validation.
(GH#216) Tasks are now stored as
TaskRecordpydantic objects which now preempts a bug introduced from providing the wrong schema.
(GH#217) Standalone QCPortal installs now report the correct version
(GH#221) Fixed a bug in
ReactionDataset.querywhere passing in
Nonewas treated as a string.
0.5.3 / 2019-03-13¶
(GH#207) All compute operations can now be augmented with a
tagwhich can be later consumed by different
QueueManagers to only carry out computations with specified tags.
(GH#210) Passwords in the database can now be generated for new users and user information can be updated (server-side only)
Collectionscan now be updated automatically from the defaults
qcfractal-managerCLI command now accepts a config file for more complex managers through Dask JobQueue. As such, many of the command line flags have been altered and can be used to either spin up a PoolExecutor, or overwrite the config file on-the-fly. As of this PR, the Dask Jobqueue component has been untested. Future updates will indicate when this has been tested.
get_Xmethods have been renamed to
query_Xto better reflect what they actually do. An exception to this is the
get_collectionsmethod which is still a true
FractalClient.list_collectionsnow respects show case sensitive results and queries are case insensitive
FractalServercan now compress responses to reduce the amount of data transmitted over the serialization. The main benefactor here is the
OpenFFWorkflowcollection which has significant transfer speed improvements due to compression.
OpenFFWorkflowcollection now has better validation on input and output data.
OpenFFWorkflowcollection only stores database
idto reduce duplication and data transfer quantities. This results in about a 50x duplication reduction.
qcfractal-templatecommand now has fields for Fractal username and password.
(GH#212) The docs for QCFractal and QCPortal have been split into separate structures. They will be hosted on separate (although linked) pages, but their content will all be kept in the QCFractal source code. QCPortal’s docs are for most users whereas QCFractal docs will be for those creating their own Managers, Fractal instances, and developers.
FractalClient.get_collectionsis now correctly case insensitive.
(GH#210) Fixed a bug in the
iteratemethod of services which returned the wrong status if everything completed right away.
reprof the MongoEngine Socket now displays correctly instead of crashing the socket due to missing attribute
0.5.2 / 2019-03-08¶
FractalClientinstances will automatically connect to the central MolSSI Fractal Server
(GH#195) Read-only access has been granted to many objects separate from their write access. This is in contrast to the previous model where either there was no access security, or everything was access secure.
(GH#197) Unknown stoichiometry are no longer allowed in the
Datasetobjects can now query keywords from aliases as well.
(GH#195) Manager cannot pull too many tasks and potentially loose data due to query limits.
Recordsnow correctly adds Provenance information
compute_torsionexample update to reflect API changes
(GH#197) Fixed an issue where CLI input flags were not correctly overwriting default values
(GH#197) Fixed an issue where
Collectionswere not correctly updating when the
savefunction was called on existing objects in the database.
_qcfractal_tagsare no longer carried through the
Recordsobjects in errant.
(GH#197) Stoichiometry information is no longer accepted in the
Datasetobject since this is not used in this class of object anymore (see
0.5.1 / 2019-03-04¶
(GH#177) Adds a new
qcfractal-templatecommand to generate
(GH#181) Pagination is added to queries, defaults to 1000 matches.
(GH#185) Begins setup documentation.
(GH#186) Begins database design documentation.
(GH#187) Results add/update is now simplified to always store entire objects rather than update partials.
(GH#189) All database compute records now go through a single
BaseRecordclass that validates and hashes the objects.
(GH#175) Refactors query massaging logic to a single function, ensures all program queries are lowercase, etc.
(GH#175) Keywords are now lazy reference fields.
(GH#182) Reworks models to have strict fields, and centralizes object hashing with many tests.
(GH#183) Centralizes duplicate checking so that accidental mixed case duplicate results could go through.
(GH#190) Adds QCArchive sphinx theme to the documentation.
0.5.0 / 2019-02-20¶
(GH#165) Separates datasets into a Dataset, ReactionDataset, and OptimizationDataset for future flexability.
(GH#168) Services now save their Procedure stubs automatically, the same as normal Procedures.
setup.pynow uses the README.md and conveys Markdown to PyPI.
(GH#171) Molecule addition now takes in a flat list and returns a flat list of IDs rather than using a dictionary.
(GH#173) Services now return their correspond Procedure ID fields.
0.5.0rc1 / 2019-02-15¶
(GH#114) A new Collection:
Generic, has been added to allow semi-structured user defined data to be built without relying only on implemented collections.
(GH#125) QCElemental common pydantic models have been integrated throughout the QCFractal code base, making a common model repository for the prevalent
Moleculeobject (and others) come from a single source. Also converted QCFractal to pass serialized pydantic objects between QCFractal and QCEngine to allow validation and (de)serialization of objects automatically.
(GH#130, GH#142, and GH#145) Pydantic serialization has been added to all REST calls leaving and entering both QCFractal Servers and QCFractal Portals. This allows automatic REST call validation and formatting on both server and client sides.
Options objects have been renamed to
KeywordSet to better match their goal (See GH#155.)
(GH#110) QCFractal now depends on QCElemental and QCEngine to improve consistent imports.
(GH#116) Queue Manger Adapters are now more generalized and inherit more from the base classes.
(GH#118) Single and Optimization procedures have been streamlined to have simpler submission specifications and less redundancy.
(GH#133) Fractal Server and Queue Manager startups are much more verbose and include version information.
(GH#135) The TorsionDriveService has a much more regular structure based on pydantic models and a new TorsionDrive model has been created to enforce both validation and regularity.
Task``s in the Mongo database can now be referenced by multiple ``Resultsand
Procedures(i.e. a single
Proceduredoes not have ownership of a
(GH#147) Service submission has been overhauled such that all services submit to a single source. Right now, only one service can be submitted at a time (to be expanded in a future feature.) TorsionDrive can now have multiple molecule inputs.
(GH#149) Package import logic has been reworked to reduce the boot-up time of QCFractal from 3000ms at the worst to about 600ms.
KeywordSetobjects are now modeled much more consistently through pydantic models and are consistently hashed to survive round trip serialization.
(GH#155) Adding multiple
Resultobjects to the database at the same time now always return their Database ID’s if added, and order of returned list of ID’s matches input order. This PR also renamed
KeywordSetto properly reflect the goal of the object.
(GH#156) Memory and Number of Cores per Task can be specified when spinning up a Queue Manager and/or Queue Adapter objects. These settings are passed on to QCEngine. These must be hard-set by users and no environment inspection is done. Users may continue to choose not to set these and QCEngine will consume everything it can when it lands on a compute.
(GH#162) Services can now be saved and fetched from the database through MongoEngine with document validation on both actions.
(GH#132) Fixed MongoEngine Socket bug where calling some functions before others resulted in an error due to lack of initialized variables.
Moleculeobjects cannot be oriented once they enter the QCFractal ecosystem (after optional initial orientation.)
Moleculeobjects also cannot be oriented by programs invoked by the QCFractal ecosystem so orientation is preserved post-calculation.
(GH#146) CI environments have been simplified to make maintaining them easier, improve test coverage, and find more bugs.
(GH#158) Database addition documents in general will strip IDs from the input dictionary which caused issues from MongoEngine having a special treatment for the dictionary key “id”.
0.4.0a / 2019-01-15¶
This is the fourth alpha release of QCFractal focusing on the database backend and compute manager enhancements.
(GH#78) Migrates Mongo backend to MongoEngine.
(GH#78) Overhauls tasks so that results or procedures own a task and ID.
(GH#78) Results and procedures are now inserted upon creation, not just completion. Added a status field to results and procedures.
(GH#78) Overhauls storage API to no longer accept arbitrary JSON queries, but now pinned kwargs.
(GH#106) Compute managers now have heartbeats and tasks are recycled after a manager has not been heard from after a preset interval.
(GH#106) Managers now also quietly shutdown on SIGTERM as well as SIGINT.
0.3.0a / 2018-11-02¶
This is the third alpha release of QCFractal focusing on a command line interface and the ability to have multiple queues interacting with a central server.
(GH#72) Queues are no longer required of FractalServer instances, now separate QueueManager instances can be created that push and pull tasks to the server.
(GH#75) CLI’s have been added for the
(GH#83) The status of server tasks and services can now be queried from a FractalClient.
(GH#82) OpenFF Workflows can now add single optimizations for fragments.
(GH#74) The documentation now has flowcharts showing task and service pathways through the code.
.dataattributes are now typed and validated with pydantic.
(GH#85) The CLI has been enhanced to cover additional features such as
(GH#84) QCEngine 0.4.0 and geomeTRIC 0.9.1 versions are now compatible with QCFractal.
0.2.0a / 2018-10-02¶
This is the second alpha release of QCFractal containing architectural changes to the relational pieces of the database. Base functionality has been expanded to generalize the collection idea with BioFragment and OpenFFWorkflow collections.
(GH#58) A overview of the QCArchive project was added to demonstrate how all modules connect together.
(GH#57) OpenFFWorkflow and BioFragment collections to support OpenFF uses cases.
(GH#57) Requested compute will now return the id of the new submissions or the id of the completed results if duplicates are submitted.
(GH#67) The OpenFFWorkflow collection now supports querying of individual geometry optimization trajectories and associated data for each torsiondrive.
(GH#43) Services and Procedures now exist in the same unified table when complete as a single procedure can be completed in either capacity.
(GH#44) The backend database was renamed to storage to prevent misunderstanding of the Database collection.
(GH#47) Tests can that require an activate Mongo instance are now correctly skipped.
(GH#51) The queue now uses a fast hash index to determine uniqueness and prevent duplicate tasks.
(GH#52) QCFractal examples are now tested via CI.
(GH#53) The MongoSocket
get_generic_by_idwas deprecated in favor of
get_genericwhere an ID can be a search field.
(GH#63) The Database collection was renamed Dataset to more correctly illuminate its purpose.
(GH#65) Collection can now be aquired directly from a client via the
0.1.0a / 2018-09-04¶
This is the first alpha release of QCFractal containing the primary structure of the project and base functionality.
(GH#41) Molecules can now be queried by molecule formula
(GH#39) The server can now use SSL protection and auto-generates SSL certificates if no certificates are provided.
(GH#31) Adds authentication to the FractalServer instance.
(GH#26) Adds TorsionDrive (formally Crank) as the first service.
(GH#26) Adds a “services” feature which can create large-scale iterative workflows.
(GH#21) QCFractal now maintains its own internal queue and uses queuing services such as Fireworks or Dask only for the currently running tasks
(GH#40) Examples can now be testing through PyTest.
(GH#38) First major documentation pass.
(GH#37) Canonicalizes string formatting to the
(GH#36) Fireworks workflows are now cleared once complete to keep the active entries small.
(GH#35) The “database” table can now be updated so that database entries can now evolve over time.
(GH#32) TorsionDrive services now track all computations that are completed rather than just the last iteration.
(GH#30) Creates a Slack Community and auto-invite badge on the main readme.
(GH#24) Remove conda-forge from conda-envs so that more base libraries can be used.
Innumerable bug fixes and improvements in this alpha release.