Data architecture in Python
The Python mechanisms of a strict data architecture: Pydantic at the boundaries, enums and datetimes that keep their wire form across 3.10 and 3.11+, unions and aliases that parse what you meant, and typed facades instead of checker suppressions.
Load bitranox:coding-data-architecture alongside this skill
The language-neutral rules live there, once: the Core Principles 1-7 cited below by number, the
anti-pattern list, the Instructions checklist (Definition of Done), and the Workflow with STEP A
to STEP D and the .data_arch_violations.json state file. Every "Core Principle N", "STEP X",
"Instructions" or "Workflow" reference in this skill names that skill's text. This skill adds what
Python needs on top of it.
Pydantic-at-the-boundary is also the input-sanitization edge. For the full per-sink output escaping (parametrized SQL, HTML autoescape, shell argv) and the boundary-vs-internal-libs scope, see
bitranox:coding-input-sanitization.
Architecture Rules
The core principles in Python
- Principle 1, typed model at every boundary: parse API requests, file reads, env vars and CLI
args into a Pydantic
BaseModelon entry. - Principle 2, typed export: produce every output with
.model_dump()or.model_dump_json(). On a wire that already exists,.model_dump_json()changes bytes and exception types: read "On an existing wire,model_dump_json()is not a drop-in" below before swapping a serializer call. - Principle 3, no internal maps: no raw
dictorMappingfor structured data inside the application; a Pydantic model or a@dataclass. - Principle 4, enums for constants: for a value that crosses an external boundary as a string,
use
StrEnum(Python 3.11+) orclass X(str, Enum)so Pydantic parses and serializes it without changing the wire format; reserveIntEnumfor values that are integers on the wire. If your floor is 3.10 and you take theclass X(str, Enum)fallback, its default string form is version-dependent - read "Thestr, Enumfallback formats differently on 3.10 and 3.11+" below before you interpolate a member anywhere. - Principle 7, no compatibility shims: code uses dataclass fields, model fields and enums directly.
What to Use When
| Scenario | Use |
|---|---|
| External input/output | Pydantic BaseModel |
| Internal business logic (no serialization) | @dataclass or Pydantic |
| Need validation | Pydantic BaseModel |
| Fixed string values (string on the wire) | StrEnum / str, Enum |
| Simple value objects | NamedTuple or @dataclass |
The str, Enum fallback formats differently on 3.10 and 3.11+
enum.StrEnum is safe: str(member), format(member) and an f-string all give the VALUE on
every version that has it. The class X(str, Enum) fallback for a 3.10 floor is NOT:
class C(str, Enum):
A = "capacity"
f"{C.A}" # 3.10 -> "capacity" 3.11+ -> "C.A"
format(C.A) # 3.10 -> "capacity" 3.11+ -> "C.A"
str(C.A) # "C.A" on EVERY version
C.A.value # "capacity" on every version
Python 3.11 made __format__ follow __str__ for mixed-in enums. So a member interpolated into
a key, a filename, a log line or any hand-built payload silently changes shape across the version
matrix, and the 3.10 lane is the one that looks correct.
This is the one place the "avoid unnecessary .value access" guidance (under "Python violation
patterns") does NOT apply. Comparisons, assignments and Pydantic serialization take the member
directly; anything that builds a string by interpolation takes .value. Measured cost of
getting it wrong: an alert-state key built as f"{pool}:{category}" wrote
rpool:IssueCategory.CAPACITY instead of rpool:capacity on 3.11+, which orphans every
previously-written state entry and re-fires every open alert.
The interpolation can happen far from the member, behind a str parameter. A member passed
into a str-annotated parameter type-checks, because the member IS a str, and the log context,
event or payload builder that receives it renders it later: %s and str() give
Layer.DEFAULTS on EVERY version, an f-string on 3.11+, and only a JSON encoder writes the value.
No caller interpolated anything, so a grep for f-strings over the callers finds nothing. For every
str parameter that feeds a log, event or payload builder, grep its callers for Enum members, then
type the parameter as the Enum and take .value INSIDE the builder, so no caller can forget it.
Pin the wire form with a test that asserts the exact string, and run the whole declared version range - a single-interpreter green cannot see this.
A datetime's wire form flips at the same 3.10 / 3.11 line
datetime.fromisoformat("2026-01-01T00:00:00Z") raises ValueError: Invalid isoformat string on
3.10 and parses on 3.11+. Pydantic's model_dump_json() writes a UTC datetime with a trailing
Z, while a hand-built dict using value.isoformat() writes +00:00. So moving an output from a
hand-built dict onto a Pydantic model changes the wire format, and a 3.10 reader that parses it
back with fromisoformat breaks with no reader code changed. Wherever a datetime reaches a key, a
filename, a log line or a payload another component reads back, pin its form:
class Report(BaseModel):
checked_at: datetime
@field_serializer("checked_at")
def _wire_checked_at(self, value: datetime) -> str:
return value.isoformat() # "+00:00" on every version; 3.10 fromisoformat reads it
Then assert the exact string in a test and run the whole declared version range, as for the enum
above. Measured (pydantic 2.14.1, Python 3.10 and 3.11): without the serializer
model_dump_json() gives "2026-01-01T00:00:00Z", with it "2026-01-01T00:00:00+00:00".
A bare BaseModel field serializes as {}
Pydantic serializes a field by its DECLARED type, not the runtime value it holds. A field annotated
as the bare BaseModel base class has no fields of its own, so .model_dump() and
.model_dump_json() silently emit {} for it no matter which real subclass instance you put there
- a type checker sees a fully-annotated field and passes, and a test that asserts only the OUTER envelope keys exist stays green too, so nothing catches it short of reading the actual value:
from pydantic import BaseModel, SerializeAsAny
class Payload(BaseModel):
name: str
count: int
class Envelope(BaseModel):
kind: str
payload: BaseModel # BAD: declared type has no fields to dump
env = Envelope(kind="a", payload=Payload(name="widget", count=3))
env.model_dump() # {'kind': 'a', 'payload': {}} <- silently empty
Two fixes, and they are not interchangeable:
SerializeAsAny[BaseModel]keeps the field typed for as long as the object lives - use it when the envelope is a live object other code still reads before the final dump. It tells Pydantic to serialize the RUNTIME type polymorphically:env.model_dump()now gives{'kind': 'a', 'payload': {'name': 'widget', 'count': 3}}.- Export at the boundary - keep the field a plain mapping and call
.model_dump()on the payload yourself in the function that emits the envelope - is the better default when the envelope exists only to be dumped (a CLI's{ok, command, data, skipped}response, an API reply). That is Core Principle 5's "one dump at output, nothing in between" applied to the payload too, and it needs no extra annotation.
Do not widen either fix into SerializeAsAny[BaseModel] | dict expecting a permissive fallback: a
raw dict validates against the bare BaseModel arm (an empty base model accepts any mapping), and
the instance you get back has no working serializer, so .model_dump() raises TypeError: 'MockValSer' object is not an instance of 'SchemaSerializer' instead of degrading gracefully. Type
the field for the concrete subclasses you actually expect, and diff the emitted JSON against a
pre-change baseline whichever fix you take - the wire format is the only thing that shows an empty
payload.
That union breaks only on a RAW DICT input; a model instance builds and dumps fine, so a test that
feeds only models stays green. The MockValSer failure is a plain TypeError, not
PydanticSerializationError, so a handler catching Pydantic's error class misses it. If a payload
field must be renamed because its natural name shadows a BaseModel attribute (schema emits a
UserWarning: Field name "schema" ... shadows an attribute in parent "BaseModel"), keep the
wire key as an alias of schema_ and configure it as the next section says, so the default dump
emits schema too; passing by_alias=True at each call covers only the calls that remember it.
On an existing wire, model_dump_json() is not a drop-in for the old serializer call
Core Principles 2 and 5 point new output at model_dump_json(). When a model REPLACES a dataclass
or dict on a wire that already exists (a JSONL file, a --json CLI envelope, a cache), or you
collapse an existing json.dumps(m.model_dump(mode="json")) or
orjson.dumps(m.model_dump(mode="json")) into model_dump_json(), the bytes change. Measured on
pydantic 2.14.2 and orjson 3.13.0:
| Input | model_dump_json() |
orjson.dumps(m.model_dump(mode="json")) |
json.dumps(m.model_dump(mode="json")) |
|---|---|---|---|
| any record | compact: {"a":1,"b":2} |
compact | spaced: {"a": 1, "b": 2} |
| int beyond 64 bits | written | raises JSONEncodeError (a TypeError) |
written |
lone surrogate in a str (a non-UTF-8 env var or filename decoded with surrogateescape) |
raises PydanticSerializationError (a ValueError) |
raises JSONEncodeError (a TypeError) |
writes the escape \udce9; with ensure_ascii=False, UnicodeEncodeError once encoded to UTF-8 |
non-UTF-8 bytes in an Any field |
raises PydanticSerializationError |
raises UnicodeDecodeError |
raises UnicodeDecodeError |
float NaN |
null |
null |
NaN (not valid JSON) |
A float field also changes the wire on its own, whichever serializer you keep: given an int it
holds a float, so 30 from a dataclass is now 30.0. Check whether a producer can pass an int
before you dismiss that line of a byte diff as test noise; a fake that feeds ints puts it on every
record.
So keep the existing serializer call, with a comment saying why, unless a byte-identity test passes
first: build every record class with the edge values above plus -0.0, non-ASCII text and naive and
aware datetimes, dump each through the old call and through model_dump_json(), and compare bytes
AND exception types (an except TypeError around the old call does not catch the ValueError the
new one raises). Switch only if they match.
A wire name that differs from the field name needs three settings
When a field's wire name differs from its Python name (a reserved word such as not becoming
not_, or schema becoming schema_), the model needs all three: the first so the DEFAULT dump
uses the wire name, the other two so the field name stays usable as the constructor keyword:
from pydantic import BaseModel, ConfigDict, Field
class Not(BaseModel):
model_config = ConfigDict(serialize_by_alias=True, validate_by_name=True)
not_: Condition = Field(validation_alias="not", serialization_alias="not")
serialize_by_alias=Trueinmodel_config(pydantic 2.11 and later). Without it,model_dump()/model_dump_json()with no arguments emit the FIELD name (not_); only a call passingby_alias=Trueuses the alias. An explicitby_alias=Falsestill overrides the config. The setting resolves PER MODEL: a nested model that sets it emits its alias inside a parent that does not, and a parent that sets it does not fix a nested model that lacks it.- The split pair
validation_alias=plusserialization_alias=, never a plainalias=. Under pyright strict a plainalias="not"makes the synthesized__init__accept only the alias, soNot(not_=x)fails withNo parameter named "not_", even withvalidate_by_name=True. The split pair keepsnot_as the constructor keyword while the alias governs the wire both ways. validate_by_name=True, soNot(not_=x)also validates at RUNTIME: with avalidation_aliasand no such setting, construction by field name raisesField required. It is the 2.11+ name for this; the olderpopulate_by_name=Truedoes the same, but pydantic marks it not recommended and due for deprecation in v3.
Prove it by dumping with NO arguments and validating that output against the wire contract the
consumer reads (its JSON Schema or a recorded fixture). A round-trip test written with
by_alias=True passes whether or not the config is set, and the no-argument dump is the path a
persistence layer actually takes, so it is the one to test.
A discriminated union tags on the Enum MEMBER, not a copy of its value
When a Pydantic 2 discriminated union switches on a StrEnum field, write the tag as
Literal[MyEnum.MEMBER]. A plain value string is validated INTO the member, in lax and strict
mode alike and from validate_python and validate_json equally, so JSON arriving over the wire
needs no duplicate string literal and no coercing before-validator:
class Kind(StrEnum):
WIDGET = "widget"
GADGET = "gadget"
class Widget(BaseModel):
kind: Literal[Kind.WIDGET] # accepts the JSON value "widget"
teeth: int
class Gadget(BaseModel):
kind: Literal[Kind.GADGET]
volts: float
Payload = Annotated[Widget | Gadget, Field(discriminator="kind")]
What it refuses is the member NAME ("WIDGET"), which is the right way round: the name is an
implementation detail of the Python side and was never the wire format.
The belief that an enum literal cannot validate from JSON is what keeps a stringly-typed
duplicate alive next to the enum - Literal["widget"] beside Kind.WIDGET - and once there are
two spellings of one tag, nothing makes them move together. Probe it rather than carrying the
fear: construct the union, call validate_json with the value string, and read the result back.
A union without a discriminator is decided by validation mode, not by fit
Smart mode takes the arm that validates STRICTLY, and only falls back to lax coercion when none does. Two traps follow, and in both the dump looks right while the parse was wrong:
- A bare
dict/Mappingarm swallows its model siblings. A dict arm accepts any mapping strictly, so it wins whenever it comes first, and even LAST it wins for every input the model arm would only accept after lax coercion ("3"for anint, from Python and from JSON alike). The dict keeps every key, somodel_dump_json()emits the input document unchanged and a round-trip test comparing documents passes, while the model's field types, refusals and defaults never ran. Give the union a discriminator (or drop the dict arm) rather than reordering it. - A union of single-kind mappings mangles a mixed-key input. For
Mapping[str, V] | Mapping[MyIntEnum, V], an input holding both kinds matches no arm strictly, so the laxstrarm turns the enum keys into value strings ('20') and the refusal surfaces far downstream; pure-name and pure-member inputs pass, so single-kind tests stay green. Narrowing toMapping[str | MyIntEnum, V]fixes the parse but breaks callers under pyright strict, becauseMapping's KEY type is invariant (Type parameter "_KT@Mapping" is invariant) and a caller'sdict[str, V]ordict[MyIntEnum, V]is not assignable to it. Keep both single-kind arms and add the mixed one, named once as an alias:Mapping[str, V] | Mapping[MyIntEnum, V] | Mapping[str | MyIntEnum, V].
Test what was BUILT, not what was dumped: assert type(parsed) is Specific (the arm you meant)
for each event shape, and for a mapping field feed a MIXED-key input and assert the stored key
types. Check a narrowed annotation with typed dict VARIABLES, never inline literals: a literal is
inferred against the parameter and hides the invariance. Measured (pydantic 2.14.1, pyright
1.1.414 strict): dict | Specific returned dict for every input, Specific | dict returned
dict for {"count": "3"}; the two-arm mapping union stored the key '20' as str, the
three-arm one kept LogLevel.INFO; the narrowed annotation rejected both typed variables and
accepted the literal. A model arm with extra="allow" did NOT swallow a stricter sibling on that
version (the specific arm won for exact and lax input, in either order) - the type(parsed)
assertion is what proves it on the version you run.
AwareDatetime accepts values that have no UTC equivalent
AwareDatetime checks only that an offset is present. A value at the edge of the datetime range
whose offset carries it past that range - 0001-01-01T00:00:00+01:00, 9999-12-31T23:59:59-01:00
- parses cleanly, and the first
astimezone(timezone.utc)downstream (a domain constructor that normalises to UTC, a comparison against a UTC value) raisesOverflowError: date value out of range. That is neitherValueErrornorValidationError, so a loader that catches the boundary's refusal lets it escape as a bare traceback. Normalise to UTC INSIDE the boundary model, and turn the overflow into aValueErrorthere yourself: pydantic turns aValueError, anAssertionErroror its ownPydanticCustomErrorraised in a validator into aValidationError, and passes anOverflowErrorthrough unchanged.
def _to_utc(value: datetime) -> datetime:
try:
return value.astimezone(timezone.utc)
except OverflowError as exc:
raise ValueError("timestamp has no UTC equivalent inside the datetime range") from exc
UtcDatetime = Annotated[AwareDatetime, AfterValidator(_to_utc)]
Build the domain object from the parsed model inside the same try that turns ValidationError
into your refusal, so anything the constructor still raises is reported there rather than past it.
Test with both range edges. Measured (pydantic 2.14.2, Python 3.15): plain AwareDatetime accepted
both edge values and astimezone raised OverflowError; an AfterValidator calling astimezone
without the try let the OverflowError out of model_validate_json; with it, both edges came
back as a ValidationError of type value_error and in-range values were normalised to UTC.
A strict IntEnum field accepts the bare integer from JSON
strict=True refuses a raw 20 for an IntEnum field from Python, but from JSON it accepts the
VALUE ({"level": 20} gives Level.INFO), refuses the member NAME ("INFO"), and words every
refusal in the internal integers (Input should be 10 or 20). When the wire format is the NAME,
parse it yourself in a BeforeValidator and dump it with a field_serializer returning
value.name. The validator must let enum MEMBERS through before refusing non-strings: an
IntEnum member IS an int, so a validator that refuses every non-str also refuses the model's
own construction from domain values (Record(level=Level.INFO)), and the write path breaks while
every JSON-parsing test stays green:
def _level_by_name(value: object) -> object:
if isinstance(value, Level): # the write path passes members; IntEnum members are ints
return value
if not isinstance(value, str):
raise ValueError("level must be a level name such as 'INFO'")
try:
return Level[value]
except KeyError:
raise ValueError(f"unknown level name {value!r}") from None
Test the round trip from a domain object, not only the parse from JSON. Measured (pydantic
2.14.2): the strict model accepted {"level":20} and refused "INFO" with Input should be 10 or
20; a validator refusing every non-str refused {"level":20} correctly but also
Record(level=Level.INFO); with the member pass-through both directions held and the dump read
{"level":"INFO"}.
Converting a str parameter to an Enum changes its contract: scope it with its tests
A proper conversion of release(part: str) to release(part: ReleasePart) moves validation to
the boundary (ReleasePart(raw) where the CLI parses), so the value is checked there and not
inside the function. That changes what its existing tests can do, so before you put the item in a
"code-only, no test changes" tier of a review or plan, run grep -rn <func_name> tests/ and read
each hit:
A test asserting that an invalid value raises
ValueErrorfrom the function breaks: the raise now happens at the boundary, inReleasePart("bogus"). Keeping a validating raise inside the function to save that test re-adds the shim this skill forbids (Core Principle 7).A test passing a RAW STRING breaks where the function dispatches by identity (
part is ReleasePart.MAJOR). It keeps passing where the code compares with==, matches on the members (case ReleasePart.MAJOR:), or looks the value up in a dict or set keyed by them, because astr, Enummember equals and hashes as its value; there only the type checker flags the call site (see STEP B). Switchingisto==saves the raw-string tests but never the raises-on-invalid one.A raw string breaks every reader of
.value: a provenance record, a log key or a serializer raisesAttributeError: 'str' object has no attribute 'value', while==and dict lookups on the same field still pass. A field of a PUBLIC constructor (a dataclass or record that outside callers build) is where this ships: a caller can hold the plain value, above all one round-tripped from the library's own output. For such a field the constructor IS the boundary (Core Principle 6), so parse there - an outside caller's input parsed once at entry, not a Core Principle 7 shim, since no internal code passes or receives the plain string - on a frozen class,__post_init__withobject.__setattr__(self, "name", Layer(self.name)), which also refuses an unknown value at construction - and testcast("Layer", "user")through every path that reads the field. A CHANGELOG line saying a plain string "still works" is checked on each such path, not on the first one that passes.An unknown raw value falls through the branches of a PUBLIC function. A value-mixin Enum (
str,int,bytes) equals its raw value, so an Enum-typed parameter of a function called from untyped code still dispatches by==, and a raw value naming no member matches no branch and lands in the lastelseas if it were that member, while pyright and the tests stay green. Where a public function has a default branch, or the old code had a guard, its entry is the boundary (Core Principle 6): parse withx = TheEnum(x), which takes a member or a raw value and raisesValueErrorfor an unknown one. Do not writeif x not in TheEnum: a raw value raisesTypeErrorthere on Python 3.10 and 3.11.
If any of these kinds of test or caller exists, the item belongs in the tier that touches tests.
str, Enum wire-compatibility saves construction and lookup tests, never identity-dispatch,
raises-on-invalid or .value-reading ones.
Branch on a StrEnum (or str, Enum) PARAMETER with ==, never is. A member equals its
own string, so "quiet" == Mode.QUIET is True while "quiet" is Mode.QUIET is False. Under is,
a caller passing the raw value does not raise: it silently takes the OTHER branch and does
something plausible and wrong. The annotation stops such a caller only inside the type-checked
code, never a config value, a payload field or an untyped test. Do not fix it by raising inside
the function either: validation belongs at the boundary parse (above), and on a hot or
supervisory path an instrument must not take down the run it observes. Keep is only on a value a
parse_* returned as Member | None, where a string cannot arrive. Add a test that passes the RAW
string (cast("Mode", "quiet")) and asserts the branch it reached by something only that branch
produces (a test that accepts either branch's output stays green on the wrong one), and grep the
diff for is <Enum>. on anything reached from a parameter.
Where the Enum lives decides where the helper can live. Read [tool.importlinter] (or
.importlinter) before choosing the parameter type: when the Enum sits in a HIGHER layer than the
module holding the str-taking helper, the helper cannot import it, and an import under
if TYPE_CHECKING: breaks the layers contract too (import-linter counts it unless the project sets
exclude_type_checking_imports, which hides the break rather than fixing it). Either move the
Enum down to a layer the helper may import, or move the helper onto the Enum as a method - the
second only when no caller of the helper sits below the Enum's layer, since those callers would
then import upward. Re-run lint-imports after the move. Measured: a low-layer
errors.refused(verb: str) could not take the models layer's enum, and every caller sat above
models, so it became a method on that enum.
Anti-Patterns to Eliminate
# BAD: Dict parameter
def process(data: dict) -> dict:
return {"name": data["name"]}
# BAD: String literals for status
if user.status == "active":
# BAD: Unnecessary conversion chain
model.model_dump() # -> dict
SomeClass(**dict_data) # -> back to model
# BAD: Converting just to access fields
d = model.model_dump()
name = d["name"] # Just use model.name!
# BAD: Pydantic -> dataclass -> Pydantic
@dataclass
class Internal:
...
internal = Internal(**pydantic_model.model_dump())
output = OutputModel(**asdict(internal))
# BAD: Fixed-shape record returned as a bare tuple across a function boundary
def split_target(text: str) -> tuple[str, str | None]: # which slot is which?
...
# BAD: str parameter whose every caller passes one of a few literals
def refused(verb: str) -> str: ...
refused("add"); refused("delete")
A fixed-shape tuple record becomes a NamedTuple (class Target(NamedTuple): host: str; zone: str | None). It is still a tuple - unpacking, indexing, == against a plain tuple and
hashing all work unchanged - so a caller that unpacks or indexes keeps working. That same equality
hides the callers you did NOT migrate: Target("a", None) == ("a", None) is True, so every
existing == assertion stays green while a test or caller still passes a plain tuple into a
parameter now typed Target (and fails at runtime the moment the code reads .host). Only the
type checker reports those call sites (pyright strict: "tuple[...] is not assignable to
Target"), and a test proving the named type arrived needs type(x) is Target plus an attribute
read, never ==. A field cannot be named index or count: it overrides the tuple method and
pyright strict reports reportIncompatibleMethodOverride (use row, records). Not this pattern:
an OS sockaddr tuple
(what socket and getaddrinfo take and return), a tuple that never leaves one function (a
local gather closure), and a variable-length tuple[X, ...]. Use a frozen dataclass instead where
a swapped unpack would still run wrongly (a (column, row) result among records written row
first), or where a union must be told from a tuple[str, ...] arm: a one-string NamedTuple is that
tuple to isinstance. A str parameter is found by its
CALLERS, not its signature: grep the call sites, and when every one passes a literal from a small
fixed set, the parameter is an Enum (see "Converting a str parameter to an Enum changes its
contract").
A Model -> dict -> Model round trip that exists to RE-VALIDATE a model_copy(update=...).
model_copy(update=...) skips validation, so Model(**m.model_copy(update=u).model_dump()) is
sometimes the only thing refusing a bad update. Do not replace it with calls into pydantic
internals (__pydantic_validator__, hand-built ValidationError.from_exception_data). The public
fix is two parts: model_config = ConfigDict(revalidate_instances="always") on the model (or the
shared base of the family), and Model.model_validate(m.model_copy(update=u)). Measured on
pydantic 2.14.2 with a max_length=5 field: with the setting, the errors equal the dict round
trip's (string_too_long at ('name',)); WITHOUT it, model_validate(instance) and
model_validate(instance, from_attributes=True) both return the over-long copy unvalidated, so
the RED for this fix is removing the setting. Before setting it, check that nothing builds these
models with model_construct (such an instance is now validated wherever it is passed into a
model) and that every validator is idempotent, since instances are now validated again each time
they enter a model.
A refusal is a boundary output too: never interpolate str(ValidationError) into one, because it
hands the caller pydantic's union-arm tags, a versioned docs URL and a slice of their own input.
Emit your own sentence plus one <dotted loc>: <reason> line per error.errors() entry and keep
the exception as __cause__. When writing or reviewing any such refusal (including one whose
ValidationError arrives wrapped in a library's exception, or a union field that reports once per
arm), read references/validation-error-messages.md.
Python additions to the Instructions checklist
Add these items to the Definition of Done from Instructions step 1 of bitranox:coding-data-architecture:
Input/Output Boundaries:
- Parse all external inputs into Pydantic models at system boundaries
- Produce all outputs through Pydantic export methods (
.model_dump(),.model_dump_json())
Internal Data Handling:
- All functions/methods use typed fields from Pydantic models or dataclasses
- Use
@dataclassonly for pure internal logic with no serialization needs
Conversion Optimization:
- Eliminate Pydantic->dataclass->Pydantic conversions (use Pydantic throughout)
Exceptions - the Enum-keyed lookup table in Python: all four conditions of the generic
exception hold (read-only, keyed by the Enum itself, a fully typed value type, a pin test). In
Python, read-only takes all three of: annotated
Mapping[...] (Final[Mapping[...]] at module level), built as MappingProxyType({...}), and
nothing writes to it after it is built. The proxy is not optional: pyright strict narrows a
Final name to the plain dict it was assigned and accepts TABLE[key] = value through it, while
it rejects that write on a MappingProxyType. A fully typed value type has no Any and no
object. The pin test is assert set(TABLE) == set(TheEnum).
Verification:
- Run
make testOR pyproject.toml tools (pytest, ruff, mypy, etc.) - fix all errors until passing - Verify type checking passes and Pydantic validation works
Python violation patterns (Instructions steps 2 and 3)
Look for these beside the generic list:
- Missing Pydantic validation at input boundaries, missing Pydantic export methods at output boundaries
- Multiple
.model_dump()calls on the same object - Pydantic <-> dataclass conversions: if code converts Pydantic -> dataclass or dataclass -> Pydantic, refactor to use Pydantic throughout. Remove compatibility shims for dict/dataclasses/Pydantic (no wrapper methods, aliases, or backward-compatibility layers)
- Unnecessary dataclass wrappers when Pydantic can be used directly
- Enum usage: use
StrEnum/str, Enumfor string-valued fields andIntEnumonly for integer-valued ones. Avoid unnecessary.valueaccess or conversions - use the enum member directly in comparisons and assignments. The exception is string INTERPOLATION under thestr, Enumfallback, where.valueis required and not optional (see "Thestr, Enumfallback formats differently on 3.10 and 3.11+"). Remove compatibility shims for enums (no wrapper methods, aliases, or backward-compatibility layers)
And refactor with these rules:
- Import external data immediately into Pydantic models with validation
- Export data using Pydantic's built-in dump methods
- Prefer Pydantic over dataclass - only use
@dataclasswhen there's a clear benefit and no Pydantic conversion needed - If you see Pydantic -> dataclass -> Pydantic, eliminate the dataclass and use Pydantic throughout
- Convert dict inputs to Pydantic models as early as possible
- All functions must operate on dataclasses or Pydantic fields
Python tooling for the Workflow
What each step of the Workflow in bitranox:coding-data-architecture runs on a Python project:
INITIALIZATION step 2: read
pyproject.toml(if it exists) to identify configured tools.STEP A, the CANDIDATE SWEEP:
grep -rnE -- '-> tuple\[|: dict|-> dict'over the target files. Measured: 9 analysers called 46 of 49 modules clean, and a grep for-> tuple[found 13 fixed-shape tuple records none of them had flagged.STEP A, the analysis subagent's Python patterns (in place of
{the language skill's patterns}in the template):- dict parameters (`: dict`, `-> dict`) - dict key access (`["key"]`, `['key']`) - string literals for statuses/modes (`== "..."`, `!= "..."`) - fixed-shape records returned as bare tuples (`-> tuple[str, str | None]`); NOT an OS sockaddr, a local gather closure, or a variable-length `tuple[X, ...]` - missing Pydantic at boundaries - unnecessary Model->dict->Model conversionsSTEP B, the type checker: tests alone cannot verify this refactor. A
StrEnummember compares and hashes equal to its string value, so every call site still passing a raw string, and every assertion on it, stays green wherever the code compares by value (==,matchon the members, a dict or set lookup) rather than by identity (is), while the type checker errors on the changed signature. Measured: three agents each reported "731 passed" while pyright had 24 errors, all of them call sites still passing raw literals into newly-Enum-typed parameters. ANamedTuplereplacing a plain tuple is the same trap: it compares equal to the tuple it replaced, so tests still passing plain tuples into the newly typed parameter stay green and only the type checker lists them.STEP C, the gate:
First, check if Makefile exists with test target:
test -f Makefile && grep -q "^test:" Makefile && echo "FOUND"If "FOUND": Execute
make testnow:make testIf NO Makefile test target, read pyproject.toml and run configured tools:
# Run each tool that has a [tool.X] section in pyproject.toml: pytest # if [tool.pytest] exists ruff check . && ruff format --check . # if [tool.ruff] exists mypy . # if [tool.mypy] exists black --check . # if [tool.black] exists isort --check . # if [tool.isort] exists
STEP C item 5, the error surface in Python: a
ValidationError, or the framework error you wrap it in, is not theValueErrorthe old call raised; pydantic adds its own error-type tags and a versioned docs URL to the message. Get each baseline's source withgit archive <HEAD or tag> src | tar -x -C <tmpdir>, then run withPYTHONPATH=<tmpdir>/src, never withpython -I- isolated mode ignores PYTHONPATH, so both arms import the installed (editable) working tree and every case reads "same". Print each arm's<package>.__file__first and require it to sit under that arm's tree. Feed wrong-TYPE values beside out-of-range ones:Truefor a count,"1"for a timeout,3.0for an int. A lax-mode model accepts all three silently (Truebecomes1); onlystrict=Trueor a strict field type refuses them. Measured: a run underpython -Ireported 20 of 20 inputs "same", and only the liveness control exposed it; the harness against the last release found a request model still acceptingtimes=Trueandtimeout="1"after five analysis passes. To keep an old refusal, re-raise the model's refusal as the old exception type carrying the validator's own text (item["msg"], orstr(item["ctx"]["error"])for avalue_errorentry; neverstr(exc); one line per entry when there are several - seereferences/validation-error-messages.md).STEP D, the candidate greps:
-> dict,: dict,-> tuple[in signatures.: dictalso matches the permitted small local dicts and Enum-keyed lookup tables and annotations that are not signatures at all, and-> tuple[matches the tuple exclusions under "Anti-Patterns to Eliminate". A module-level table annotatedMapping[...]orFinal[dict[...]]matches none of those patterns, so also grep^[A-Za-z_][A-Za-z0-9_]*: .*(dict|Mapping)\[and judge each hit the same way. A type alias is not a table; skip it.
Example Transformations
Before (violation):
def process_user(data: dict) -> dict:
if data["status"] == "active":
return {"name": data["name"], "role": "member"}
After (compliant):
from enum import StrEnum
from pydantic import BaseModel
class UserStatus(StrEnum):
ACTIVE = "active"
INACTIVE = "inactive"
class UserRole(StrEnum):
MEMBER = "member"
ADMIN = "admin"
class UserInput(BaseModel):
name: str
status: UserStatus
class UserOutput(BaseModel):
name: str
role: UserRole
def process_user(data: UserInput) -> UserOutput:
if data.status == UserStatus.ACTIVE:
return UserOutput(name=data.name, role=UserRole.MEMBER)
Define the types; never suppress or exclude the checker
The discipline that forbids stringly-typed values also forbids silencing the type checker. When pyright/mypy strict flags your code - very often a third-party stub gap, not a real defect - the completion path is to DEFINE the missing types, not to suppress the diagnostic or exclude the file.
Order of preference:
- Add the real annotation or generic argument.
- Wrap a partially-typed third-party symbol in a typed facade you define - a
Protocolplus acast, a factory whose return type is theProtocol(no cast; see the filelock example), or a local.pyistub onstubPath- so call sites see complete types. - Only if neith
…(truncated)