for developers

pw.persistence

This page provides the documentation on the classes required to set up persistence. See persistence articles for the introduction to the topic.

Configuration classes

class pw.persistence.Backend(engine_data_storage, fs_path=None)

[source]

The settings of a backend, which is used to persist the computation state.

classmethod filesystem(path)

sourceConfigure the filesystem backend.

  • Parameters
    path (str | PathLike[str]) – the path to the root directory in the file system, which will be used to store the persisted data.
  • Returns
    Class instance denoting the filesystem storage backend with root directory at path.

classmethod s3(root_path, bucket_settings)

sourceConfigure the S3 backend.

  • Parameters
    • root_path (str) – path to the root in the S3 storage, which will be used to store persisted data;
    • bucket_settings (AwsS3Settings) – the settings for S3 bucket connection in the same format as they are used by S3 connectors.
  • Returns
    Class instance denoting the S3 storage backend with root directory as root_path and connection settings given by bucket_settings.

class pw.persistence.Config(backend, *, snapshot_interval_ms=0, snapshot_access=<pathway.engine.SnapshotAccess object>, persistence_mode=<pathway.engine.PersistenceMode object>, continue_after_replay=True)

[source]

Configure the data persistence. An instance of this class should be passed as a parameter to pw.run in case persistence is enabled.

  • Parameters
    • backend (Backend) – persistence backend configuration;
    • snapshot_interval_ms (int) – the desired duration between snapshot updates in milliseconds;

classmethod simple_config(backend, snapshot_interval_ms=0, snapshot_access=api.SnapshotAccess.FULL, persistence_mode=api.PersistenceMode.PERSISTING, continue_after_replay=True)

sourceConstruct config from a single instance of the Backend class, using this backend to persist metadata and snapshot.

Note that this method is deprecated and is left for the backward compatibility purposes only. Please use the pw.persistence.Config constructor instead.

  • Parameters
    • backend (Backend) – storage backend settings;
    • snapshot_interval_ms – the desired freshness of the persisted snapshot in milliseconds. The greater the value is, the more the amount of time that the snapshot may fall behind, and the less computational resources are required.
  • Returns
    Persistence config.