Skip to content

Core API

The examples import these objects from arti when they are re-exported there. The reference entries use canonical module paths so the generated docs can resolve them reliably.

arti.graphs.Graph

Bases: Model

Graph stores a web of Artifacts connected by Producers.

name instance-attribute

artifacts = Field(default_factory=ArtifactBox) class-attribute instance-attribute

backend = Field(default_factory=_get_memory_backend, exclude=True) class-attribute instance-attribute

path_tags = frozendict() class-attribute instance-attribute

artifact_to_key property

artifact_producer_outputs property

dependencies cached property

producers cached property

producer_outputs cached property

build(executor=None)

snapshot(*, connection=None)

Identify a "unique" ID for this Graph at this point in time.

The ID aims to encode the structure of the Graph plus a snapshot of the raw Artifact data (partition kinds and contents). Any change that would affect data should prompt an ID change, however changes to this ID don't directly cause data to be reproduced.

NOTE: There is currently a gap (and thus race condition) between when the Graph ID is computed and when we read raw Artifacts data during Producer builds.

read(artifact, *, annotation=None, connection=None, snapshot=None, storage_partition_snapshots=None, view=None) 

read(
    artifact: Artifact,
    *,
    annotation: type[T],
    connection: BackendConnection | None = None,
    snapshot: GraphSnapshot | None = None,
    storage_partition_snapshots: StoragePartitionSnapshots
    | None = None,
    view: None = None,
) -> T

read(
    artifact: Artifact,
    *,
    annotation: Any | None = None,
    connection: BackendConnection | None = None,
    snapshot: GraphSnapshot | None = None,
    storage_partition_snapshots: StoragePartitionSnapshots
    | None = None,
    view: View | None = None,
) -> Any

write(data, *, artifact, input_fingerprint=None, partition_key=PartitionKey(), view=None, snapshot=None, connection=None)

arti.graphs.GraphSnapshot

Bases: Model

GraphSnapshot represents the state of a Graph and the referenced raw data at a point in time.

GraphSnapshot encodes the structure of the Graph plus a snapshot of the raw Artifact data (partition kinds and contents) at a point in time. Any change that would affect data should prompt an ID change.

id instance-attribute

graph instance-attribute

artifacts property

backend property

name property

from_graph(graph, *, connection=None) classmethod

Snapshot the Graph and all existing raw data.

NOTE: There is currently a gap (and thus race condition) between when the Graph ID is computed and when we read raw Artifact data during Producer builds.

build(executor=None)

tag(tag, *, overwrite=False, connection=None)

from_tag(name, tag, *, connectable) classmethod

read(artifact, *, annotation=None, connection=None, storage_partition_snapshots=None, view=None) 

read(
    artifact: Artifact,
    *,
    annotation: type[T],
    connection: BackendConnection | None = None,
    storage_partition_snapshots: StoragePartitionSnapshots
    | None = None,
    view: None = None,
) -> T

read(
    artifact: Artifact,
    *,
    annotation: Any | None = None,
    connection: BackendConnection | None = None,
    storage_partition_snapshots: StoragePartitionSnapshots
    | None = None,
    view: View | None = None,
) -> Any

write(data, *, artifact, connection=None, input_fingerprint=None, partition_key=PartitionKey(), view=None)

arti.artifacts.Artifact

Bases: Model

An Artifact is the base structure describing an existing or generated dataset.

An Artifact is comprised of three key elements: - type: spec of the data's structure, such as data types, nullable, etc. - format: the data's serialized format, such as CSV, Parquet, database native, etc. - storage: the data's persistent storage system, such as blob storage, database native, etc.

In addition to the core elements, an Artifact can be tagged with additional annotations (to associate it with human knowledge) and statistics (to track derived characteristics over time).

type instance-attribute

format = Field(default_factory=Format.get_default, validate_default=True) class-attribute instance-attribute

storage = Field(default_factory=Storage.get_default, validate_default=True) class-attribute instance-attribute

annotations = () class-attribute instance-attribute

statistics = () class-attribute instance-attribute

producer_output = None class-attribute instance-attribute

cast(value) classmethod

Attempt to convert an arbitrary value to an appropriate Artifact instance.

Artifact.cast is used to convert values assigned to an ArtifactBox (such as Graph.artifacts) into an Artifact. When called with: - an Artifact instance, it is returned - a Producer instance with a single output Artifact, the output Artifact is returned - a Producer instance with a multiple output Artifacts, an error is raised - other types, we attempt to map to a Type and return an Artifact instance with defaulted Format and Storage

arti.fingerprints.Fingerprint

Bases: int

Fingerprint represents a unique identity as an int64 value.

Using an int(64) has a number of convenient properties: - can be combined independent of order with XOR - can be stored relatively cheaply - 0 acts as an "identity" value when combined (5 ^ 0 = 5) - is relatively cross-platform (across databases, languages, etc)

is_identity property

combine(*others)

from_int(x) classmethod

from_string(x) classmethod

Fingerprint an arbitrary string.

Fingerprints using Farmhash Fingerprint64, converted to int64 via two's complement.

identity() classmethod

Return a Fingerprint that, when combined, will return the other Fingerprint.

arti.partitions.PartitionKey

Bases: frozendict[str, PartitionField]

The set of named PartitionFields that uniquely identify a single partition.

types_from(type_) classmethod

arti.producers.producer(*, annotations=None, map=None, name=None, validate_outputs=None, version=None)

arti.producers.one_to_one(input_name)

Create a map function where each output partition depends on one input partition.

Output partitions reuse the input_name partition keys. input_name must be the Producer's only build input (enforced when the Producer class is created).

arti.io.read(type_, format, storage_partition_snapshots, view)

arti.io.write(data, type_, format, storage_partition, view)