zarr_metadata.definition

zarr_metadata.definition

Reading metadata fields against their definitions, in either Zarr format.

Every extension point of a Zarr document (a v3 codec, data type, chunk grid, chunk key encoding or storage transformer; a v2 dtype or numcodecs codec) is a metadata field: a name, and a configuration whose JSON the extension defines. A definition declares what that JSON is and what it may contain. A definition is a value, not a class to subclass. It has:

• name, the name the metadata uses;
• configuration, the TypedDict for the configuration's JSON. This is the single declaration of the JSON: the type checker is compiled from it, and a checked configuration has it as its static type. It is interpreted as the typing spec defines a TypedDict: total, Required, NotRequired, closed and extra_items mean what they mean to a type checker, whether or not its module postpones annotations. A member's type carries its bounds, in the same way pydantic reads them: a gzip level is Annotated[int, Interval(ge=0, le=9)];
• rules, a function over that TypedDict that yields the spec violations a type cannot express (such as constraints between members), located in the configuration.

The kind of metadata a definition defines is its type. The v3 kinds are in zarr_metadata.v3.definition, and the v2 kinds are in zarr_metadata.v2.definition. This module holds what both formats share: Definition and WithFillValue, which the kinds derive from; resolve, which reads a field in a scope; the three results of a reading; and Context, the protocol of a scope.

Reading JSON. There are three steps. Each feeds the next, and each can be used on its own by a caller that has only JSON:

1. check(value, SomeTypedDict), from zarr_metadata.typed_json, type-checks JSON against a TypedDict and needs nothing else. It returns a value of the TypedDict or None, and every problem, each with a location. The value contains only what the TypedDict admits. A member typed with a field alias is checked as metadata field JSON.
2. definition.read_configuration(configuration) returns the configuration as that definition reads it. It type-checks the configuration, checks the envelope of each nested field (an extra member, a must_understand of false), and then runs the rules, for one configuration: GZIP_CODEC.read_configuration({"level": 12}).

3. resolve(field, CodecDefinition, CORE_AND_EXTENSIONS) reads a whole field in a scope. It checks the envelope, maps the name to a definition, checks the configuration, and reads each nested field the same way. It returns the result of the reading and every problem. The result is one of three types. AcceptedField is a field read by the definition that claims its name, with the configuration it checked and allowed, and its nested fields, each read the same way. UnclaimedField is a name that nothing in scope claims. It is not checked, which is what keeps the format open. RefusedField is a field with problems that explain why it was rejected. Each type has the field's JSON, the name it is written with, the kind it was read as (read_as), the definition that claims its name (None for UnclaimedField), and its nested fields as the scope read them (nested). The two types a model can hold, AcceptedField and UnclaimedField, have a configuration and to_json(), which returns the field as a document writes it. Two of them are equal when they read the same, however each was spelled. ResolvedField is the union of the three, for use with match. configuration_of(field, GZIP_CODEC) returns the configuration of a field that GZIP_CODEC accepted, typed as its TypedDict (GzipCodecConfiguration), and None for any other field, including one that GZIP_CODEC claimed and refused. A field is read as one of the kinds of its format, with or without type arguments. resolve(field, Definition, context) raises TypeError, because nothing is filed under Definition. A whole v3 array document is read by read_array_metadata_v3, in zarr_metadata.model. Its reading returns each field with its location in the document.

   from zarr_metadata.definition import resolve from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS) resolved # RefusedField(..., definition=CodecDefinition(name='gzip'), ...) problems[0].loc # ('configuration', 'level') problems[0].input # 12 dict(problems[0].ctx) # {'ge': 0, 'le': 9}

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 5}}, CodecDefinition, CORE_AND_EXTENSIONS) resolved.definition is GZIP_CODEC # True resolved.configuration # {'level': 5}

Problems are values, not exceptions: ValidationProblem(loc, message, kind), with kind one of invalid_type, invalid_value, missing_key, unknown_key and invalid_json. A field with an unknown key is still read: the key is reported, and the configuration is checked without it. So a consumer that tolerates unknown keys can filter problems by kind and use the result. The field is valid only when there are no problems at all. Each problem stores the content of its message as data, as pydantic's errors and zod's issues do: input is the value found at loc (the 12 above), and ctx is what was expected, when that is more than a type: the bounds {"ge": 0, "le": 9}, or the values of a closed set.

Writing an extension. Write a TypedDict that declares the configuration's JSON, including bounds; a function for the rules a type cannot express; and a definition. Then build a scope that contains the definition. The TypedDict must be a typing_extensions.TypedDict, because closed and extra_items come from PEP 728, and typing.TypedDict does not accept them on the Python versions this package supports. The rules receive the configuration and its nested fields as the scope read them. A field that is read stores its nested fields as AcceptedField.nested, a Nested mapping keyed by location, so a struct's rules can access its field types. read_configuration reads in no scope, so it passes no nested fields. A rule's message should show a value as JSON, as the package's own messages do, with zarr_metadata.problems.shown: null, [1, 2], "C". A rule reports the location of a problem. The value at that location becomes the problem's input automatically. Define each function at module top level. A model holds the definitions that read its fields, so the model can be pickled, and compares equal after loading, only if its definitions can. A lambda or a closure cannot be pickled, and a functools.partial can be pickled but does not compare equal to its loaded copy.

from collections.abc import Iterator
from typing import Annotated, NotRequired

from annotated_types import Ge
from typing_extensions import TypedDict

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition


class AcmeLz4Configuration(TypedDict, closed=True):
    acceleration: Annotated[int, Ge(1)]
    dictionary: NotRequired[str]
    dictionary_size: NotRequired[Annotated[int, Ge(1)]]


def acme_lz4_rules(
    configuration: AcmeLz4Configuration, nested: Nested
) -> Iterator[ValidationProblem]:
    if "dictionary" in configuration and "dictionary_size" not in configuration:
        yield ValidationProblem(
            ("dictionary_size",), "a dictionary needs its size", "missing_key"
        )


ACME_LZ4 = CodecDefinition(
    name="acme.lz4",
    configuration=AcmeLz4Configuration,
    kind="bytes_bytes",
    size="dynamic",
    rules=acme_lz4_rules,
)
CONTEXT = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), ACME_LZ4])

The TypedDict declares how keys it does not declare are handled. With closed=True, such a key is a problem, as above. With extra_items=, such a key must hold that type. With closed=False, anything is allowed. A TypedDict that declares none of these is open by default and would accept a misspelled key silently, so a definition rejects it. Its members must be JSON shapes: int, float, bool, str, None, JSONValue, a Literal, tuple[T, ...] and tuple[T1, T2], a union, a TypedDict, Mapping[str, V], a NewType and a type alias. A number's type may carry bounds, written with annotated-types and read the way pydantic reads them: Gt, Ge, Lt, Le and Interval, at most one from each side, at any depth. For example, tuple[Annotated[int, Ge(1)], ...] bounds each element. A value out of bounds is an invalid_value problem. Its message says what the type admits ("expected an integer >= 1, got 0"), and its ctx holds the bounds. The rules run only on a configuration within its bounds, so a rule can rely on them, as pydantic's after-validators and zod's refinements do. Until a value out of bounds is fixed, it is the only problem reported for the configuration. Annotated may also carry a note, as a string or a Doc. Any other metadata (a MinLen, a Predicate, pydantic's Field) is rejected when the definition is built, because a type whose constraint the checker does not enforce would give false information.

A data type also declares its fill value. fill_value is the JSON shape of a fill value, as an annotation the checker reads (for example Int8FillValue, whose type carries the range). fill_value_rules is a function that yields the spec violations in a fill value of that shape that the type cannot express, such as a hex string of the wrong width, or a number of byte values that does not match the size. The rules receive the configuration, the nested fields as the scope read them, and the typed fill value, so a struct checks each field's fill value against that field's own type. fill_value_problems(data_type, value) checks a fill value against a data type field the scope read. If nothing in scope claims the data type, the fill value is not checked. A data type that does not declare its fill value accepts any JSON. A fill value can have more than one spelling ("NaN" and "0x7fc00000" are the same float32), so a data type declares the canonical spelling of each value with fill_value_canonical. It receives the same arguments as the rules, plus a fill value the rules allow. Two fill values are the same value of the type exactly when their canonical spellings are written the same way: the same JSON, as json.dumps writes it. == is not the same test, because it treats -0.0 (a distinct float32) as equal to 0.0. canonical_fill_value(data_type, value) returns the canonical spelling, or UNSET for a fill value with a problem. A data type that does not declare fill_value_canonical keeps each value as written.

The simplest spelling. A field without problems has a simplest equivalent spelling, and fields are compared by it. Each nested field is written in its own simplest spelling, then the definition's canonical is applied (blosc drops a typesize that noshuffle ignores, and a rectilinear grid run-length encodes its chunk shapes). The envelope uses the shortest form every reader accepts: a data type with nothing to configure is its bare name, and any other field is an object, {"name": ...}, because a Zarr v3.0 reader does not accept a short-hand name in codecs. Raw bits write their size back into the name, in decimal, so r008 becomes r8. A field with any problem, including an unknown key, has no simplest spelling, because a simpler spelling would erase what its author wrote. The result of canonical is checked again, and an invalid result raises ValueError, which indicates a bug in the definition. canonical_fill_value writes a fill value in the same way, using the data type that read it.

Scopes as values. A scope reads documents of one Zarr format. ZarrV3Context, in zarr_metadata.v3.definition, and ZarrV2Context, in zarr_metadata.v2.definition, are the scopes of each format. They are separate classes, and Context is the protocol both implement: the Zarr format a scope reads, and claimant, which says which definition of a kind reads a name. That is all resolve needs, so any object that provides these two can be a scope. A scope has one slot for each kind of field its format has, named like the document member that holds such fields: ZarrV3Context(codecs=[...], data_types=[...]) builds one, and context.codecs["gzip"] reads one slot. A type checker rejects a definition in the wrong slot, and a scope of one format given to a reader of the other. The constructor and the reader also reject them at run time, with TypeError. Two scopes are equal when they are of one format and file the same definitions, and equal scopes have equal hashes. context.union(*others) returns the smallest scope that contains each of them, or raises a ScopeConflictError naming each name filed with two different definitions. update replaces the definitions in the slots it is given and keeps the others, and takes a slot as definitions or as a mapping by name, so context.update(codecs={**context.codecs, "gzip": MY_GZIP}) replaces one definition on purpose. A model's refined_in moves it to a scope that claims more names and contradicts nothing. refines orders two models by how much they know. A name that was unclaimed and is now read by a definition is a gain. The reverse is a loss. One name read by two different definitions is a conflict.

A definition checks itself when it is built. Each of these raises a TypeError that says what is wrong: a configuration that is not a TypedDict, that does not declare how undeclared keys are handled, or that has a member no checker reads (the error names the TypedDict that contains it); a name that is not a string; a member declared as a function (rules, canonical, fill_value_rules, storage, shape_rules, chunk_lengths, chunk_rules, transition, pipelines) that is not a function; a data type's fill_value that no checker reads, or that contains a metadata field, which a value of the data type never does; a codec kind that is not one of the three, or a size that is not "static" or "dynamic"; a function that is never called for a codec of its kind (chunk rules or pipelines of a bytes -> bytes codec, which receives bytes, or a transition of a codec that outputs bytes); a data type named the way raw bits of one size are written; an extra field named after a member the spec defines, whose finite is not a boolean, or whose configuration does not declare must_understand. Building a definition files it nowhere: only a scope built from it does.

ClaimKey module-attribute

ClaimKey: TypeAlias = tuple[type[Definition[Any]], str]

A kind and the name a definition is filed under. A scope's claimant looks up definitions by this key.

Nested module-attribute

Nested: TypeAlias = Mapping[Loc, ResolvedField[Any]]

The fields in a configuration, each as the scope read it, keyed by location in the configuration.

ResolvedField module-attribute

ResolvedField = TypeAliasType(
    "ResolvedField",
    "AcceptedField[D] | UnclaimedField | RefusedField[D]",
    type_params=(D,),
)

One metadata field as a scope read it: accepted by the definition that claims its name, unclaimed, or refused.

AcceptedField dataclass

Bases: Generic[D]

A field that a definition in scope read: the name it is written with, the definition, and the configuration the definition allowed.

A problem with the envelope (an extra member, a must_understand of false) is reported with the field, and the field is still read. The same applies to a problem in a nested field, which belongs to that nested field, as nested shows. Two fields are equal when they read the same, however each was spelled, as compared by field_key. For example, "bytes" and {"name": "bytes"} are one field. So are a blosc with and without the typesize that noshuffle ignores, because the definition's canonical removes it. Equal fields have equal hashes.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class AcceptedField(Generic[D]):
    """A field that a definition in scope read: the name it is written with, the definition, and the configuration the definition allowed.

    A problem with the envelope (an extra member, a `must_understand` of
    `false`) is reported with the field, and the field is still read. The
    same applies to a problem in a nested field, which belongs to that
    nested field, as `nested` shows. Two fields are equal when they read
    the same, however each was spelled, as compared by `field_key`. For
    example, `"bytes"` and `{"name": "bytes"}` are one field. So are a
    blosc with and without the `typesize` that `noshuffle` ignores, because
    the definition's `canonical` removes it. Equal fields have equal
    hashes.
    """

    json: JSONValue
    """The field as written, refined (arrays as tuples). It is not used for equality."""
    name: str
    """The name the field is written with, for example `"r16"`, even though its definition is filed under `r*`."""
    definition: D
    """The definition that read the field."""
    configuration: Mapping[str, JSONValue]
    """The configuration, type-checked and allowed by the rules. For raw bits, this is the configuration encoded in the name.

    Each nested field is written as that field's `to_json` writes it, so
    the configuration shows what was read, however it was spelled. For
    example, a shard's `"crc32c"` and `{"name": "crc32c"}` are the same
    index codec.
    """
    nested: Nested = dataclasses.field(default_factory=_nothing_nested)
    """The fields in the configuration, each as the scope read it, keyed by location in the configuration.

    For example, a struct's field types are at `("fields", 0, "data_type")`
    and a shard's codecs are at `("codecs", 0)`. A definition's functions
    use this to inspect the fields inside its own configuration.
    """
    read_as: type[Definition[Any]] = dataclasses.field(init=False, repr=False)
    """The kind of metadata the field was read as: the kind of its definition."""

    def __eq__(self, other: object) -> bool:
        if not is_field(other):
            return NotImplemented
        return field_key(self) == field_key(other)

    def __hash__(self) -> int:
        return hash(field_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. A field built by hand (an
        # extension may do this) fails here, not later in a function that
        # trusts it.
        definition = cast("object", self.definition)
        kind = kind_of(definition)
        refusal = (
            f"a field read is read by a definition of a kind, got {definition!r}"
            if kind is None
            else _misread(definition, kind, self.name)
        )
        if refusal is not None:
            raise TypeError(refusal)
        object.__setattr__(self, "read_as", kind)
        # Everything the field returns is read-only at every level, so a
        # field cannot reach a state where its key, `==` and `refines`
        # disagree.
        object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(self, "configuration", frozen(self.configuration))
        object.__setattr__(self, "nested", MappingProxyType(dict(self.nested)))

    def __reduce__(self) -> tuple[Callable[..., AcceptedField[Any]], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the field from its constructor arguments.
        return _accepted_field, (
            copied(self.json),
            self.name,
            self.definition,
            copied(self.configuration),
            dict(self.nested),
        )

    def to_json(self) -> JSONValue:
        """The field as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the field.

        The envelope uses the shortest form every reader accepts. A data
        type with nothing to configure is written as its bare name, as core
        data types have been written since Zarr v3.0. Any other field is an
        object, `{"name": ...}`, because a Zarr v3.0 reader does not accept a
        bare name in `codecs`
        (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592).
        A name that encodes its configuration, as the raw bits name does, is
        written alone.
        """
        return copied(written_json(self))

configuration instance-attribute

configuration: Mapping[str, JSONValue]

The configuration, type-checked and allowed by the rules. For raw bits, this is the configuration encoded in the name.

Each nested field is written as that field's to_json writes it, so the configuration shows what was read, however it was spelled. For example, a shard's "crc32c" and {"name": "crc32c"} are the same index codec.

definition instance-attribute

definition: D

The definition that read the field.

json instance-attribute

json: JSONValue

The field as written, refined (arrays as tuples). It is not used for equality.

name instance-attribute

name: str

The name the field is written with, for example "r16", even though its definition is filed under r*.

nested class-attribute instance-attribute

nested: Nested = dataclasses.field(
    default_factory=_nothing_nested
)

The fields in the configuration, each as the scope read it, keyed by location in the configuration.

For example, a struct's field types are at ("fields", 0, "data_type") and a shard's codecs are at ("codecs", 0). A definition's functions use this to inspect the fields inside its own configuration.

read_as class-attribute instance-attribute

read_as: type[Definition[Any]] = dataclasses.field(
    init=False, repr=False
)

The kind of metadata the field was read as: the kind of its definition.

to_json

to_json() -> JSONValue

The field as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the field.

The envelope uses the shortest form every reader accepts. A data type with nothing to configure is written as its bare name, as core data types have been written since Zarr v3.0. Any other field is an object, {"name": ...}, because a Zarr v3.0 reader does not accept a bare name in codecs (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592). A name that encodes its configuration, as the raw bits name does, is written alone.

Source code in src/zarr_metadata/definition/_definition.py

def to_json(self) -> JSONValue:
    """The field as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the field.

    The envelope uses the shortest form every reader accepts. A data
    type with nothing to configure is written as its bare name, as core
    data types have been written since Zarr v3.0. Any other field is an
    object, `{"name": ...}`, because a Zarr v3.0 reader does not accept a
    bare name in `codecs`
    (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592).
    A name that encodes its configuration, as the raw bits name does, is
    written alone.
    """
    return copied(written_json(self))

Conflict dataclass

One place where two readings of a name disagree: the key, the definition one reading claimed, the definition the other found, and the location in the document when known.

Source code in src/zarr_metadata/definition/_scope.py

@dataclass(frozen=True, slots=True)
class Conflict:
    """One place where two readings of a name disagree: the key, the definition one reading claimed, the definition the other found, and the location in the document when known."""

    key: ClaimKey
    claimed: Definition[Any] | None
    found: Definition[Any] | None
    loc: Loc | None = None
    written: str | None = None
    """The name the document uses at the conflict location, for example `r16` for a key of `r*`; None when no document is available."""

    def __str__(self) -> str:
        kind, filed = self.key
        name = filed if self.written is None else self.written
        where = "" if self.loc is None else f" at {self.loc!r}"
        claimed, found = definition_said(self.claimed), definition_said(self.found)
        if claimed == found:
            # Nothing a description shows tells them apart, such as their
            # rules, so say that they differ.
            return f"{kind_name(kind)} {name!r}{where}: two different definitions, each {claimed}"
        return f"{kind_name(kind)} {name!r}{where}: claimed {claimed}, found {found}"

written class-attribute instance-attribute

written: str | None = None

The name the document uses at the conflict location, for example r16 for a key of r*; None when no document is available.

Context

Bases: Protocol

A scope: the definitions in scope while metadata of one Zarr format is read.

This is what reading a field needs from a scope: the Zarr format of the documents it reads, and which definition of a kind claims a name. ZarrV3Context and ZarrV2Context implement it, each with one slot per kind of field its format has, and so does any other object that provides these two. resolve reads a field in any Context.

Source code in src/zarr_metadata/definition/_definition.py

class Context(Protocol):
    """A scope: the definitions in scope while metadata of one Zarr format is read.

    This is what reading a field needs from a scope: the Zarr format of the
    documents it reads, and which definition of a kind claims a name.
    `ZarrV3Context` and `ZarrV2Context` implement it, each with one slot
    per kind of field its format has, and so does any other object that
    provides these two. `resolve` reads a field in any `Context`.
    """

    @property
    def zarr_format(self) -> Literal[2, 3]: ...

    def claimant(self, kind: type[D], name: str) -> D | None: ...

Definition dataclass

Bases: Generic[C]

One extension's metadata as JSON: its name, its configuration TypedDict, and its rules.

configuration is the TypedDict, and so it is the single declaration of the JSON. The checker is compiled from it, it is the static type of a checked configuration, and a document's author writes to it. It is interpreted as the typing spec defines a TypedDict: total, Required, NotRequired, closed and extra_items mean what they mean to a type checker. It also declares how keys it does not declare are handled. With closed=True, such a key is a problem, reported and left out. With extra_items=, such a key must have that type. With closed=False, anything is allowed. rules yields each violation of the spec it finds in a configuration of that type. It receives only a configuration that has passed the check: it contains only what the TypedDict admits, each member is within the bounds its type declares, and the nested fields are passed as the scope read them (for example, a struct's field types). The nested fields are empty when no scope read the configuration. read_configuration runs both steps, for a caller that has JSON.

Each function receives the configuration as a read-only view, not a dict. copy.deepcopy and json.dumps reject this view. A function that simplifies a spelling must build a new mapping, such as {**configuration} without the member, instead of editing its input.

canonical is where two spellings of the configuration that mean the same thing are reduced to one.

A definition built by hand rejects inputs it could not read with: a configuration that is not a TypedDict, that does not declare how undeclared keys are handled, or that has a member no checker reads (the error names the member); and a name or rules of the wrong type.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class Definition(Generic[C]):
    """One extension's metadata as JSON: its name, its configuration TypedDict, and its rules.

    `configuration` is the TypedDict, and so it is the single declaration
    of the JSON. The checker is compiled from it, it is the static type of
    a checked configuration, and a document's author writes to it. It is
    interpreted as the typing spec defines a TypedDict: `total`,
    `Required`, `NotRequired`, `closed` and `extra_items` mean what they
    mean to a type checker. It also declares how keys it does not declare
    are handled. With `closed=True`, such a key is a problem, reported and
    left out. With `extra_items=`, such a key must have that type. With
    `closed=False`, anything is allowed. `rules` yields each violation of
    the spec it finds in a configuration of that type. It receives only a
    configuration that has passed the check: it contains only what the
    TypedDict admits, each member is within the bounds its type declares,
    and the nested fields are passed as the scope read them (for example, a
    struct's field types). The nested fields are empty when no scope read
    the configuration. `read_configuration` runs both steps, for a caller
    that has JSON.

    Each function receives the configuration as a read-only view, not a
    `dict`. `copy.deepcopy` and `json.dumps` reject this view. A function
    that simplifies a spelling must build a new mapping, such as
    `{**configuration}` without the member, instead of editing its input.

    `canonical` is where two spellings of the configuration that mean the
    same thing are reduced to one.

    A definition built by hand rejects inputs it could not read with: a
    `configuration` that is not a TypedDict, that does not declare how
    undeclared keys are handled, or that has a member no checker reads
    (the error names the member); and a `name` or rules of the wrong type.
    """

    _is_kind: ClassVar[bool] = False
    """Whether this class is a kind, which is what a scope files definitions by. Set by `class Kind(Definition, kind=True, format=3)`.

    Read from the class's own namespace and never inherited: a subclass of
    a kind is a definition of that kind, and does not declare `kind`.
    """
    _format: ClassVar[Literal[2, 3] | None] = None
    """The Zarr format of the documents that contain fields of this kind. It is declared with the kind and inherited by its definitions, and it sets the format a scope reads."""
    label: ClassVar[str] = "definition"
    """The name of the kind in error messages, for example "codec"."""
    field_aliases: ClassVar[tuple[TypeAliasType, ...]] = ()
    """The field aliases used to annotate a configuration member that holds a field of this kind. For a codec: `ZarrV3CodecField` and `ZarrV3StaticCodecField`."""

    name: str
    """The name the metadata uses. A scope files the definition under this name."""
    configuration: type[C]
    """The configuration TypedDict."""
    rules: Callable[[C, Nested], Iterable[ValidationProblem]] = no_rules
    """The spec violations in a well-typed configuration and its nested fields, located in the configuration."""
    canonical: Callable[[C], C] = unchanged
    """A well-typed, allowed configuration in its simplest equivalent spelling.

    This handles only the definition's own members. `canonicalize` puts
    each nested field in its own canonical form, because it knows the
    location of each one.
    """

    @classmethod
    def _name_problem(cls, name: str, at: Loc) -> ValidationProblem | None:
        """The problem to report at `at` when no document of the format may use `name` for a field of this kind; None when it may. For Zarr v3, a name may be used when it is a valid extension name under the spec."""
        return name_problem(name, at)

    @classmethod
    def _well_named(cls, name: str) -> bool:
        """Whether a document of the format may use `name` for a field of this kind, as decided by `_name_problem`."""
        return cls._name_problem(name, ()) is None

    @classmethod
    def _filed_under(cls, name: str) -> tuple[str | None, dict[str, JSONValue] | None]:
        """Interpret a name used in a document: return the name its definition is filed under, and the configuration encoded in the name.

        By default a name is filed as itself and encodes nothing:
        `(name, None)`. A kind whose names encode configuration overrides
        this. For example, the v3 data type `r16` is filed under `r*` with
        `{"bits": 16}`. A name that no document uses, and that exists only
        to file a definition, returns `(None, None)`.
        """
        return name, None

    def _carrying_name(self, configuration: Mapping[str, JSONValue]) -> str | None:
        """The name that encodes `configuration` for this definition. This is the inverse of `filed_under`: `r16` for `r*` with `{"bits": 16}`. None when this definition's names encode nothing."""
        return None

    @classmethod
    def _named_configuration(
        cls, value: object
    ) -> tuple[str | None, Mapping[str, object] | None, Problems]:
        """Split `value` (a field as a document of the format writes it) into `(name, configuration, problems)`, as the module-level `named_configuration` does for a v3 field."""
        return named_configuration(value)

    @classmethod
    def _envelope_problems(cls, value: object) -> Problems:
        """Every reason `value` is not a valid envelope for a field of this kind in this format. The contents of the configuration are not checked."""
        return envelope_problems(value, allow_must_understand_false=False)

    @classmethod
    def _envelope_json(cls, name: str, configuration: Mapping[str, JSONValue]) -> JSONValue:
        """A field with `name` and `configuration`, as a document of the format writes it, in the shortest form every reader accepts. For v3 this is an object."""
        if len(configuration) != 0:
            return {"name": name, "configuration": configuration}
        return {"name": name}

    @classmethod
    def _unclaimed_json(
        cls, json: JSONValue, name: str, configuration: Mapping[str, JSONValue]
    ) -> JSONValue:
        """A field of this kind that no definition claims, as a document writes it: `json` is the field as written. By default the name and configuration go in the envelope every reader accepts, as for a field a definition read."""
        return cls._envelope_json(name, configuration)

    @classmethod
    def _refined(
        cls, data: object, loc: Loc, definition: Definition[Any] | None
    ) -> tuple[JSONValue | None, Problems]:
        """`data`, a field of this kind at `loc`, refined to JSON before it is read, with every reason it is not JSON. `definition` is the definition that claims the field, if any. By default every number must be finite, as RFC 8259 requires."""
        return refine_json(data, loc)

    @classmethod
    def _configuration_loc(cls, loc: Loc) -> Loc:
        """The location of the configuration of a field at `loc`: under `configuration` for v3, or at the field itself for a format that writes the parameters next to the name."""
        return (*loc, "configuration")

    @classmethod
    def _name_loc(cls, loc: Loc) -> Loc:
        """The location of the name of a field at `loc` that is written as an object: under `name` for v3."""
        return (*loc, "name")

    @classmethod
    def _alias_problems(
        cls, alias: TypeAliasType, definition: Definition[Any], loc: Loc
    ) -> Problems:
        """The problems with using `definition` in a member annotated with `alias`, one of this kind's `field_aliases`. Empty by default.

        A kind with an alias that admits only some of its definitions
        overrides this. For example, `ZarrV3StaticCodecField` admits only
        codecs of static size.
        """
        return ()

    def __init_subclass__(
        cls, *, kind: bool = False, format: Literal[2, 3] | None = None, **kwargs: object
    ) -> None:
        """Register a subclass. `kind=True` declares a kind, with the Zarr `format` of the documents that contain its fields. This marks the class but not its subclasses, as `typing.Protocol` and SQLAlchemy's `__abstract__` do."""
        # Use the explicit form, not bare `super()`. A dataclass with slots
        # is rebuilt, and the cell that bare `super()` reads refers to the
        # discarded class.
        super(Definition, cls).__init_subclass__(**kwargs)
        # A dataclass with slots is built twice. The second build has no
        # class keywords but reuses the first class's namespace, and the
        # last class built is the one used to read documents. So the mark is
        # kept in the namespace, and the aliases are registered last.
        if kind:
            if format not in (2, 3):
                msg = (
                    f"{cls.__name__}: a kind declares the Zarr format its fields belong to, "
                    f"format=2 or format=3, got {format!r}"
                )
                raise TypeError(msg)
            cls._is_kind = True
            cls._format = format
        elif format is not None:
            msg = f"{cls.__name__}: only a kind declares a format; a definition of a kind has its kind's"
            raise TypeError(msg)
        for alias in cls.__dict__.get("field_aliases", ()):
            _FIELD_KINDS[alias] = cls

    def __post_init__(self) -> None:
        refusal = _malformed(self) or self._refusal()
        if refusal is not None:
            raise TypeError(refusal)
        kind = kind_of(self)
        try:
            _vet(self.configuration, None if kind is None else format_of(kind))
        except TypeError as error:
            msg = f"{self.name!r}: {error}"
            raise TypeError(msg) from error

    def __repr__(self) -> str:
        # Keep this short, because readings that contain definitions show
        # them. A full repr would show every function the definition holds,
        # with its address.
        return f"{type(self).__name__}(name={self.name!r})"

    def _refusal(self) -> str | None:
        """A description of what is wrong with the members a kind adds; None if nothing is wrong, or if the kind adds no members."""
        return None

    def _finite_numbers(self) -> bool:
        """Whether every number in a configuration this definition reads must be finite, as RFC 8259 requires. A kind whose configuration may hold user data overrides this."""
        return True

    @property
    def requires_configuration(self) -> bool:
        """Whether a document must include a configuration, meaning the TypedDict has a required key."""
        return len(typeddict_keys(self.configuration).required) != 0

    def _check_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """Type-check `value` as this definition's configuration, and check the envelope of each nested field.

        `zarr_metadata.typed_json.check` does only the type check. This
        method also checks the envelope of each metadata field that a
        member holds.
        """
        configuration, problems = _configuration_checked(
            value, self.configuration, loc, finite=self._finite_numbers()
        )
        return configuration, with_input(problems, value, loc)

    def read_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """Type-check `value`, then check it with the rules. Return the configuration if it is valid, and every problem.

        The rules run only on a configuration that passed the type check,
        including its bounds, and whose nested fields are well formed. Such
        a configuration contains only what its TypedDict admits. So a rule
        never receives a member of the wrong type, a member out of bounds,
        or a member the type does not allow. No scope reads the nested
        fields here, so the rules see none of them as read. A rule about a
        nested field (for example, a struct field whose type has values of
        varying size) finds nothing to check. `resolve` reads the field in a
        scope and runs every rule.
        """
        configuration, problems = self._check_configuration(value, loc)
        if configuration is None:
            return None, problems
        refused = ruled(self, lambda: self.rules(read_only(configuration), _nothing_nested()), loc)
        return (configuration if len(refused) == 0 else None), (
            *problems,
            *with_input(refused, value, loc),
        )

canonical class-attribute instance-attribute

canonical: Callable[[C], C] = unchanged

A well-typed, allowed configuration in its simplest equivalent spelling.

This handles only the definition's own members. canonicalize puts each nested field in its own canonical form, because it knows the location of each one.

configuration instance-attribute

configuration: type[C]

The configuration TypedDict.

field_aliases class-attribute

field_aliases: tuple[TypeAliasType, ...] = ()

The field aliases used to annotate a configuration member that holds a field of this kind. For a codec: ZarrV3CodecField and ZarrV3StaticCodecField.

label class-attribute

label: str = 'definition'

The name of the kind in error messages, for example "codec".

name instance-attribute

name: str

The name the metadata uses. A scope files the definition under this name.

requires_configuration property

requires_configuration: bool

Whether a document must include a configuration, meaning the TypedDict has a required key.

rules class-attribute instance-attribute

rules: Callable[
    [C, Nested], Iterable[ValidationProblem]
] = no_rules

The spec violations in a well-typed configuration and its nested fields, located in the configuration.

__init_subclass__

__init_subclass__(
    *,
    kind: bool = False,
    format: Literal[2, 3] | None = None,
    **kwargs: object,
) -> None

Register a subclass. kind=True declares a kind, with the Zarr format of the documents that contain its fields. This marks the class but not its subclasses, as typing.Protocol and SQLAlchemy's __abstract__ do.

Source code in src/zarr_metadata/definition/_definition.py

def __init_subclass__(
    cls, *, kind: bool = False, format: Literal[2, 3] | None = None, **kwargs: object
) -> None:
    """Register a subclass. `kind=True` declares a kind, with the Zarr `format` of the documents that contain its fields. This marks the class but not its subclasses, as `typing.Protocol` and SQLAlchemy's `__abstract__` do."""
    # Use the explicit form, not bare `super()`. A dataclass with slots
    # is rebuilt, and the cell that bare `super()` reads refers to the
    # discarded class.
    super(Definition, cls).__init_subclass__(**kwargs)
    # A dataclass with slots is built twice. The second build has no
    # class keywords but reuses the first class's namespace, and the
    # last class built is the one used to read documents. So the mark is
    # kept in the namespace, and the aliases are registered last.
    if kind:
        if format not in (2, 3):
            msg = (
                f"{cls.__name__}: a kind declares the Zarr format its fields belong to, "
                f"format=2 or format=3, got {format!r}"
            )
            raise TypeError(msg)
        cls._is_kind = True
        cls._format = format
    elif format is not None:
        msg = f"{cls.__name__}: only a kind declares a format; a definition of a kind has its kind's"
        raise TypeError(msg)
    for alias in cls.__dict__.get("field_aliases", ()):
        _FIELD_KINDS[alias] = cls

read_configuration

read_configuration(
    value: object, loc: Loc = ()
) -> tuple[C | None, Problems]

Type-check value, then check it with the rules. Return the configuration if it is valid, and every problem.

The rules run only on a configuration that passed the type check, including its bounds, and whose nested fields are well formed. Such a configuration contains only what its TypedDict admits. So a rule never receives a member of the wrong type, a member out of bounds, or a member the type does not allow. No scope reads the nested fields here, so the rules see none of them as read. A rule about a nested field (for example, a struct field whose type has values of varying size) finds nothing to check. resolve reads the field in a scope and runs every rule.

Source code in src/zarr_metadata/definition/_definition.py

def read_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
    """Type-check `value`, then check it with the rules. Return the configuration if it is valid, and every problem.

    The rules run only on a configuration that passed the type check,
    including its bounds, and whose nested fields are well formed. Such
    a configuration contains only what its TypedDict admits. So a rule
    never receives a member of the wrong type, a member out of bounds,
    or a member the type does not allow. No scope reads the nested
    fields here, so the rules see none of them as read. A rule about a
    nested field (for example, a struct field whose type has values of
    varying size) finds nothing to check. `resolve` reads the field in a
    scope and runs every rule.
    """
    configuration, problems = self._check_configuration(value, loc)
    if configuration is None:
        return None, problems
    refused = ruled(self, lambda: self.rules(read_only(configuration), _nothing_nested()), loc)
    return (configuration if len(refused) == 0 else None), (
        *problems,
        *with_input(refused, value, loc),
    )

EmptyConfiguration

Bases: TypedDict

The configuration of a definition with nothing to configure. Its field is written with its name only.

Source code in src/zarr_metadata/definition/_definition.py

class EmptyConfiguration(TypedDict, closed=True):
    """The configuration of a definition with nothing to configure. Its field is written with its name only."""

RefusedField dataclass

Bases: Generic[D]

A field that could not be read: it is not a field at all, not JSON, or rejected by the definition that claims its name. Its problems explain why.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class RefusedField(Generic[D]):
    """A field that could not be read: it is not a field at all, not JSON, or rejected by the definition that claims its name. Its problems explain why."""

    json: JSONValue | UNSET
    """The field as written, refined (arrays as tuples). `UNSET` when it is not JSON, which no document can contain."""
    name: str | None
    """The name the field is written with; None when it has no name."""
    read_as: type[Definition[Any]]
    """The kind of metadata the field was read as."""
    definition: D | None = None
    """The definition that claims the name and rejected the field; None when nothing in scope claims it, or when the field has no name."""
    nested: Nested = dataclasses.field(default_factory=_nothing_nested)
    """The nested fields, each as the scope read it; empty when the configuration was not checked against its TypedDict."""

    def __eq__(self, other: object) -> bool:
        if not is_field(other):
            return NotImplemented
        return field_key(self) == field_key(other)

    def __hash__(self) -> int:
        return hash(field_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. Remove type arguments from
        # `read_as`, as `resolve` does.
        kind = as_kind(self.read_as)
        object.__setattr__(self, "read_as", kind)
        definition = cast("object", self.definition)
        refusal = None if definition is None else _misread(definition, kind, self.name)
        if refusal is not None:
            raise TypeError(refusal)
        if self.json is not UNSET:
            object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(self, "nested", MappingProxyType(dict(self.nested)))

    def __reduce__(self) -> tuple[Callable[..., RefusedField[Any]], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the field from its constructor arguments.
        return _refused_field, (
            UNSET if self.json is UNSET else copied(self.json),
            self.name,
            self.read_as,
            self.definition,
            dict(self.nested),
        )

definition class-attribute instance-attribute

definition: D | None = None

The definition that claims the name and rejected the field; None when nothing in scope claims it, or when the field has no name.

json instance-attribute

json: JSONValue | UNSET

The field as written, refined (arrays as tuples). UNSET when it is not JSON, which no document can contain.

name instance-attribute

name: str | None

The name the field is written with; None when it has no name.

nested class-attribute instance-attribute

nested: Nested = dataclasses.field(
    default_factory=_nothing_nested
)

The nested fields, each as the scope read it; empty when the configuration was not checked against its TypedDict.

read_as instance-attribute

read_as: type[Definition[Any]]

The kind of metadata the field was read as.

ScopeConflictError

Bases: ValueError

Raised when two scopes, or a scope and a reading, give one name two different meanings, or when one would lose a meaning the other has.

Every conflict is stored in .conflicts, in the same way that MetadataValidationError stores every problem.

Source code in src/zarr_metadata/definition/_scope.py

class ScopeConflictError(ValueError):
    """Raised when two scopes, or a scope and a reading, give one name two different meanings, or when one would lose a meaning the other has.

    Every conflict is stored in `.conflicts`, in the same way that
    `MetadataValidationError` stores every problem.
    """

    def __init__(self, conflicts: Sequence[Conflict]) -> None:
        self.conflicts: tuple[Conflict, ...] = tuple(conflicts)
        super().__init__("; ".join(str(conflict) for conflict in self.conflicts))

    def __reduce__(
        self,
    ) -> tuple[type[ScopeConflictError], tuple[tuple[Conflict, ...]], dict[str, object]]:
        # Pickle and copy the exception as it was raised, keeping its notes
        # and attributes. The default exception reduce calls the constructor
        # with the message, not the conflicts.
        return type(self), (self.conflicts,), dict(self.__dict__)

UnclaimedField dataclass

A field that nothing in scope claims: an extension that the scope does not check. Accepting such fields is what keeps the format open.

It is equal to another unclaimed field when both are of the same kind, have the same name, and a document writes the same JSON for both. Nothing in scope interprets the field, so it is compared as JSON text, as field_key defines.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class UnclaimedField:
    """A field that nothing in scope claims: an extension that the scope does not check. Accepting such fields is what keeps the format open.

    It is equal to another unclaimed field when both are of the same kind,
    have the same name, and a document writes the same JSON for both.
    Nothing in scope interprets the field, so it is compared as JSON text,
    as `field_key` defines.
    """

    json: JSONValue
    """The field as written, refined (arrays as tuples). It is not used for equality."""
    name: str
    """The name that nothing in scope claims."""
    read_as: type[Definition[Any]]
    """The kind of metadata the field was read as: the kind a claiming definition would have."""
    configuration: Mapping[str, JSONValue] = dataclasses.field(init=False)
    """The configuration as written, not checked; empty when no configuration is written."""

    def __eq__(self, other: object) -> bool:
        if not is_field(other):
            return NotImplemented
        return field_key(self) == field_key(other)

    def __hash__(self) -> int:
        return hash(field_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. Remove type arguments from
        # `read_as`, as `resolve` does.
        object.__setattr__(self, "read_as", as_kind(self.read_as))
        name = cast("object", self.name)
        if not isinstance(name, str) or not self.read_as._well_named(name):
            msg = (
                "a field nothing in scope claims is named as a document names a "
                f"{self.read_as.label}, got {name!r}"
            )
            raise TypeError(msg)
        _, written, _ = self.read_as._named_configuration(self.json)
        configuration: Mapping[str, object] = {} if written is None else written
        object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(
            self, "configuration", frozen(cast("Mapping[str, JSONValue]", configuration))
        )

    def __reduce__(self) -> tuple[Callable[..., UnclaimedField], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the field from its constructor arguments.
        return _unclaimed_field, (copied(self.json), self.name, self.read_as)

    @property
    def definition(self) -> None:
        """The definition that read the field. Always None, because no definition read it."""
        return None

    @property
    def nested(self) -> Nested:
        """The nested fields as the scope read them. Always empty, because nothing read the configuration."""
        return _nothing_nested()

    def to_json(self) -> JSONValue:
        """The field as a document writes it, sharing no objects with the field. The configuration is as written, in the envelope form every reader accepts, as `AcceptedField.to_json` writes it."""
        return copied(written_json(self))

configuration class-attribute instance-attribute

configuration: Mapping[str, JSONValue] = dataclasses.field(
    init=False
)

The configuration as written, not checked; empty when no configuration is written.

definition property

definition: None

The definition that read the field. Always None, because no definition read it.

json instance-attribute

json: JSONValue

The field as written, refined (arrays as tuples). It is not used for equality.

name instance-attribute

name: str

The name that nothing in scope claims.

nested property

nested: Nested

The nested fields as the scope read them. Always empty, because nothing read the configuration.

read_as instance-attribute

read_as: type[Definition[Any]]

The kind of metadata the field was read as: the kind a claiming definition would have.

to_json

to_json() -> JSONValue

The field as a document writes it, sharing no objects with the field. The configuration is as written, in the envelope form every reader accepts, as AcceptedField.to_json writes it.

Source code in src/zarr_metadata/definition/_definition.py

def to_json(self) -> JSONValue:
    """The field as a document writes it, sharing no objects with the field. The configuration is as written, in the envelope form every reader accepts, as `AcceptedField.to_json` writes it."""
    return copied(written_json(self))

WithFillValue dataclass

Bases: Definition[C]

A definition of a type whose arrays have a fill value: the fill value's JSON shape, the rules for it, and its canonical spelling.

This is not a kind. The data type kind of each format derives from it, so fill_value_problems can check a fill value for either format.

fill_value is the JSON shape of a fill value, as an annotation. The checker reads it the same way it reads a configuration's members, including any range. fill_value_rules yields the spec violations in a fill value of that shape that the type cannot express. It receives the configuration, the nested fields as the scope read them, and the typed fill value. fill_value_canonical writes a fill value without problems in the single canonical spelling of its value. So two fill values are the same value of the type exactly when their canonical spellings are written the same way.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True, repr=False)
class WithFillValue(Definition[C]):
    """A definition of a type whose arrays have a fill value: the fill value's JSON shape, the rules for it, and its canonical spelling.

    This is not a kind. The data type kind of each format derives from it,
    so `fill_value_problems` can check a fill value for either format.

    `fill_value` is the JSON shape of a fill value, as an annotation. The
    checker reads it the same way it reads a configuration's members,
    including any range. `fill_value_rules` yields the spec violations in
    a fill value of that shape that the type cannot express. It receives
    the configuration, the nested fields as the scope read them, and the
    typed fill value. `fill_value_canonical` writes a fill value without
    problems in the single canonical spelling of its value. So two fill
    values are the same value of the type exactly when their canonical
    spellings are written the same way.
    """

    fill_value: object = JSONValue
    """The JSON shape of a fill value, as an annotation, such as `Int8FillValue`."""
    fill_value_rules: Callable[[C, Nested, Any], Iterable[ValidationProblem]] = no_rules
    """The spec violations in a fill value of that shape, located in the fill value."""
    fill_value_canonical: Callable[[C, Nested, Any], JSONValue] = fill_value_as_written
    """A fill value without problems, in the single canonical spelling of its value."""

    def _refusal(self) -> str | None:
        try:
            _fill_value_parser(self.fill_value)
        except TypeError as error:
            return f"{self.name!r}: fill_value: {error}"
        return None

fill_value class-attribute instance-attribute

fill_value: object = JSONValue

The JSON shape of a fill value, as an annotation, such as Int8FillValue.

fill_value_canonical class-attribute instance-attribute

fill_value_canonical: Callable[
    [C, Nested, Any], JSONValue
] = fill_value_as_written

A fill value without problems, in the single canonical spelling of its value.

fill_value_rules class-attribute instance-attribute

fill_value_rules: Callable[
    [C, Nested, Any], Iterable[ValidationProblem]
] = no_rules

The spec violations in a fill value of that shape, located in the fill value.

canonical_fill_value

canonical_fill_value(
    data_type: ResolvedField[F], value: object
) -> JSONValue | UNSET

value (a fill value of data_type, a data type field a scope read) in the single canonical spelling of its value; UNSET when it has a problem.

The spelling comes from the data type's fill_value_canonical. Two fill values of a data type are the same value exactly when their canonical spellings are written the same way: the same JSON as json.dumps writes it. == is not the same test, because it treats -0.0 as equal to 0.0. A fill value with a problem found by fill_value_problems has no canonical spelling, just as canonical_of returns none for a field with a problem. The result is UNSET and not None, because None is the JSON null, which is a valid fill value for a data type the scope did not read (such a data type keeps a fill value as written).

Source code in src/zarr_metadata/definition/_definition.py

def canonical_fill_value(data_type: ResolvedField[F], value: object) -> JSONValue | UNSET:
    """`value` (a fill value of `data_type`, a data type field a scope read) in the single canonical spelling of its value; `UNSET` when it has a problem.

    The spelling comes from the data type's `fill_value_canonical`. Two
    fill values of a data type are the same value exactly when their
    canonical spellings are written the same way: the same JSON as
    `json.dumps` writes it. `==` is not the same test, because it treats
    `-0.0` as equal to `0.0`. A fill value with a problem found by
    `fill_value_problems` has no canonical spelling, just as `canonical_of`
    returns none for a field with a problem. The result is `UNSET` and not
    `None`, because `None` is the JSON `null`, which is a valid fill value
    for a data type the scope did not read (such a data type keeps a fill
    value as written).
    """
    if len(fill_value_problems(data_type, value)) != 0:
        return UNSET
    refined, _ = refine_json(value, ())
    return spelled_canonically(data_type, refined)

canonical_of

canonical_of(
    resolved: ResolvedField[Any],
    problems: Sequence[ValidationProblem],
) -> JSONValue | None

resolved (a field a scope read) in its simplest equivalent spelling, as canonicalize writes it; None when it has a problem.

problems are the field's problems, as returned by resolve, or as with_problems returns them for each field of a reading. Only a field with no problems has a simplest spelling. A simpler spelling of a field with a problem would erase what its author wrote (such as a key its TypedDict does not declare), or would write something invalid. The result is built from what the scope read, without reading the field again, and is plain JSON that shares nothing with the field.

Source code in src/zarr_metadata/definition/_definition.py

def canonical_of(
    resolved: ResolvedField[Any], problems: Sequence[ValidationProblem]
) -> JSONValue | None:
    """`resolved` (a field a scope read) in its simplest equivalent spelling, as `canonicalize` writes it; None when it has a problem.

    `problems` are the field's problems, as returned by `resolve`, or as
    `with_problems` returns them for each field of a reading. Only a field
    with no problems has a simplest spelling. A simpler spelling of a
    field with a problem would erase what its author wrote (such as a key
    its TypedDict does not declare), or would write something invalid.
    The result is built from what the scope read, without reading the
    field again, and is plain JSON that shares nothing with the field.
    """
    if len(problems) != 0:
        return None
    simplest = _simplest(resolved)
    return None if simplest is None else copied(simplest)

configuration_of

configuration_of(
    resolved: ResolvedField[Any],
    definition: Definition[C],
) -> C | None

The configuration of resolved, typed as definition declares it, if definition accepted it.

AcceptedField stores its configuration as a read-only mapping. When called with the definition that read the field, or one equal to it (as with a pickled field), this returns a copy of it, a plain dict typed as the TypedDict, which shares nothing with the field. Returns None for any other field: one that another definition read, one that no definition claims, and one that definition claimed and refused.

Source code in src/zarr_metadata/definition/_definition.py

def configuration_of(resolved: ResolvedField[Any], definition: Definition[C]) -> C | None:
    """The configuration of `resolved`, typed as `definition` declares it, if `definition` accepted it.

    `AcceptedField` stores its configuration as a read-only mapping. When
    called with the definition that read the field, or one equal to it
    (as with a pickled field), this returns a copy of it, a plain `dict`
    typed as the TypedDict, which shares nothing with the field. Returns
    None for any other field: one that another definition read, one that
    no definition claims, and one that `definition` claimed and refused.
    """
    if not isinstance(resolved, AcceptedField) or resolved.definition != definition:
        return None
    return cast("C", copied(resolved.configuration))

fill_value_problems

fill_value_problems(
    data_type: ResolvedField[F],
    value: object,
    loc: Loc = (),
) -> Problems

The problems with value as a fill value of data_type (a data type field a scope read).

value is refined to JSON first. If it is not JSON, that is reported first, whatever the data type. It is then checked against the JSON shape that the data type's definition declares, and by the fill value rules, in the same way read_configuration reads a configuration: a key the shape does not declare is reported and left out, and the rules still check the rest. The rules see the nested fields of the configuration as the scope read them, so a struct checks each field's fill value against that field's own type. If the scope did not read the data type (it is out of scope or invalid), a JSON fill value is not checked. loc prefixes every problem.

Source code in src/zarr_metadata/definition/_definition.py

def fill_value_problems(data_type: ResolvedField[F], value: object, loc: Loc = ()) -> Problems:
    """The problems with `value` as a fill value of `data_type` (a data type field a scope read).

    `value` is refined to JSON first. If it is not JSON, that is reported
    first, whatever the data type. It is then checked against the JSON
    shape that the data type's definition declares, and by the fill value
    rules, in the same way `read_configuration` reads a configuration: a
    key the shape does not declare is reported and left out, and the rules
    still check the rest. The rules see the nested fields of the
    configuration as the scope read them, so a struct checks each field's
    fill value against that field's own type. If the scope did not read
    the data type (it is out of scope or invalid), a JSON fill value is not
    checked. `loc` prefixes every problem.
    """
    refined, problems = refine_json(value, loc)
    if len(problems) != 0 or not isinstance(data_type, AcceptedField):
        return with_input(problems, value, loc)
    definition, configuration = data_type.definition, data_type.configuration
    typed, problems = _fill_value_parser(definition.fill_value)(refined, loc)
    if not _usable(problems):
        return with_input(problems, value, loc)
    refused = ruled(
        definition,
        lambda: definition.fill_value_rules(read_only(configuration), data_type.nested, typed),
        loc,
    )
    return with_input((*problems, *refused), value, loc)

resolve

resolve(
    data: object,
    kind: type[D],
    context: Context,
    loc: Loc = (),
) -> tuple[ResolvedField[D], Problems]

Read data (one metadata field) as a kind in context. Return the result of the reading, and every problem.

This runs all three steps for one field. data is refined to JSON and its envelope is checked. An extra member, a configuration that is not an object, and a must_understand that is not a boolean or is false are each a problem. The name is mapped to a definition in context. The configuration is checked against its TypedDict and by its rules. Each nested field found by the check is read the same way, in the same scope. A problem in a nested field belongs to that field and is reported at its location, as with a document's fields. The result is an AcceptedField read by the definition that claims the name; an UnclaimedField when nothing in scope claims the name (an extension with no model, not checked, which is what keeps the format open); or a RefusedField, with the problems that explain why. loc prefixes every problem. kind must be one of the kinds of a format (for v3, those ZarrV3Kind names; for v2, those ZarrV2Kind names), with or without type arguments, and context a scope of that format. Anything else raises TypeError.

Source code in src/zarr_metadata/definition/_definition.py

def resolve(
    data: object, kind: type[D], context: Context, loc: Loc = ()
) -> tuple[ResolvedField[D], Problems]:
    """Read `data` (one metadata field) as a `kind` in `context`. Return the result of the reading, and every problem.

    This runs all three steps for one field. `data` is refined to JSON and
    its envelope is checked. An extra member, a `configuration` that is not
    an object, and a `must_understand` that is not a boolean or is `false`
    are each a problem. The name is mapped to a definition in `context`.
    The configuration is checked against its TypedDict and by its rules.
    Each nested field found by the check is read the same way, in the same
    scope. A problem in a nested field belongs to that field and is
    reported at its location, as with a document's fields. The result is
    an `AcceptedField` read by the definition that claims the name; an
    `UnclaimedField` when nothing in scope claims the name (an extension
    with no model, not checked, which is what keeps the format open); or a
    `RefusedField`, with the problems that explain why. `loc` prefixes
    every problem. `kind` must be one of the kinds of a format (for v3,
    those `ZarrV3Kind` names; for v2, those `ZarrV2Kind` names), with or
    without type arguments, and `context` a scope of that format. Anything
    else raises `TypeError`.
    """
    if not _is_scope(context):
        msg = f"a field is read in a scope, such as CORE_AND_EXTENSIONS, got {described(context)}"
        raise TypeError(msg)
    asked = as_kind(kind)
    _check_format(asked, context)
    name = asked._named_configuration(data)[0]
    definition = (
        None if name is None or not asked._well_named(name) else context.claimant(asked, name)
    )
    refined, problems = asked._refined(data, loc, definition)
    return resolve_refined(data, refined, problems, kind, context, loc)

with_problems

with_problems(
    fields: Iterable[tuple[Loc, ResolvedField[Any]]],
    problems: Sequence[ValidationProblem],
) -> Iterator[tuple[Loc, ResolvedField[Any], Problems]]

Yield each of fields with its location and the problems in problems located inside it, including inside its nested fields.

fields and problems must come from the same read: a reading's fields() and problems, or fields_of a field and the problems that resolve returned with it. A field's problems include those found when it was read, and those the document found with it in its position (its place in the pipeline, the chunk it receives, the array's shape). So a field with no problems is valid in that position, and canonical_of can write it. This groups the problems at every depth, as zod's treeifyError groups issues: a problem with a shard's inner codec belongs to the inner codec and to the shard. Each field comes before its nested fields, in the order of fields_of, so the last field that lists a problem is the innermost field containing it. A problem outside every field (with the fill value or the shape) is not listed under any field.

Source code in src/zarr_metadata/definition/_definition.py

def with_problems(
    fields: Iterable[tuple[Loc, ResolvedField[Any]]], problems: Sequence[ValidationProblem]
) -> Iterator[tuple[Loc, ResolvedField[Any], Problems]]:
    """Yield each of `fields` with its location and the problems in `problems` located inside it, including inside its nested fields.

    `fields` and `problems` must come from the same read: a reading's
    `fields()` and `problems`, or `fields_of` a field and the problems that
    `resolve` returned with it. A field's problems include those found when
    it was read, and those the document found with it in its position (its
    place in the pipeline, the chunk it receives, the array's shape). So a
    field with no problems is valid in that position, and `canonical_of`
    can write it. This groups the problems at every depth, as zod's
    `treeifyError` groups issues: a problem with a shard's inner codec
    belongs to the inner codec and to the shard. Each field comes before
    its nested fields, in the order of `fields_of`, so the last field that
    lists a problem is the innermost field containing it. A problem
    outside every field (with the fill value or the shape) is not listed
    under any field.
    """
    located = list(fields)
    held: dict[Loc, list[ValidationProblem]] = {loc: [] for loc, _ in located}
    for found in problems:
        for depth in range(len(found.loc) + 1):
            holder = held.get(found.loc[:depth])
            if holder is not None:
                holder.append(found)
    for loc, field in located:
        yield loc, field, tuple(held[loc])