Django Structured Fields

Django Structured JSON Field — Agent Skill

1. Project Identity

2. Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                        Django Model Layer                              │
│  StructuredJSONField(JSONField)  ←──  schema=SomePydanticModel         │
│  StructuredDescriptior           (lazy validation on __get__)          │
└────────────┬───────────────────────────────────────┬────────────────────┘
             │ validates via                         │ formfield()
             ▼                                       ▼
┌────────────────────────────┐       ┌──────────────────────────────────┐
│   Pydantic Layer           │       │   Widget Layer (Admin)           │
│   BaseModel (BaseModelMeta)│       │   StructuredJSONFormWidget       │
│   ForeignKey[T]            │       │   StructuredJSONFormField        │
│   QuerySet[T]              │       │   template: widget.html          │
│   FieldSerializer          │       │   JSONEditor + Select2           │
└────────────┬───────────────┘       └──────────────────────────────────┘
             │ @model_validator                       ▲ AJAX search
             ▼                                        │
┌────────────────────────────┐       ┌──────────────────────────────────┐
│   Cache Layer              │       │   Views                          │
│   CacheEngine              │       │   search_view (staff-only)       │
│   Cache / ThreadSafeCache  │       │   /structured_field/search_model │
│   ValueWithCache           │       └──────────────────────────────────┘
│   RelInfo                  │
└────────────────────────────┘       ┌──────────────────────────────────┐
                                     │   DRF Contrib                    │
                                     │   StructuredModelSerializer      │
                                     │   StructuredJSONField (DRF)      │
                                     │   FieldsOverrideMixin            │
                                     └──────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│                    Frontend (reactive-forms/)                           │
│   main.js → JSONEditor init    callbacks.js → Select2 AJAX             │
│   patches.js → Select2 editor   scss/ → Admin theme integration        │
│   Rollup build → structured/static/js/ & css/                          │
└─────────────────────────────────────────────────────────────────────────┘

3. Directory Map & File Roles

Python — structured/

Frontend — reactive-forms/

Build output → structured/static/

Template

Tests — tests/

4. Key Concepts & Design Patterns

4.1 Lazy Descriptor Validation

StructuredDescriptior (in fields.py) extends DeferredAttribute. Data stays as a raw dict in instance.__dict__ until the attribute is accessed. On first access, schema.validate_python() converts it to a Pydantic model instance and caches in __dict__. This means zero validation cost if the field is never read.

4.2 Metaclass-Driven Annotation Rewriting

BaseModelMeta.__new__ runs patch_annotation() on every field annotation. This auto-converts:

• A bare Django Model class → ForeignKey[Model]
• QuerySet[Model] → Annotated[QuerySet[Model], Field(default_factory=...)]
• Forward references are resolved via module globals + parent frame namespace

This allows users to write author: User and get full FK behavior automatically.

4.3 Batched Cache Loading (N+1 Prevention)

CacheEngine.build_cache() flow:

1. get_all_fk_data(data) walks all nested fields recursively, collecting {Model: [pk_list]}.
2. _build_plainset() deduplicates PKs per model class.
3. _populate_cache() issues one Model.objects.filter(pk__in=pks) per model.
4. _set_cache_values() replaces raw PKs in the data dict with ValueWithCache wrappers.
5. fetch_cache() calls .retrieve() on each wrapper after validation.

Relationship types tracked via RelInfo:

• fk — single ForeignKey
• qs — QuerySet (many)
• rel — nested BaseModel (recurse into its cache engine)
• rel_l — list of nested BaseModels

4.4 Signal-Based Cache Invalidation

Cache registers post_save and pre_delete Django signals to keep cached model instances up-to-date. ThreadSafeCache (CACHE.SHARED = True) is a singleton — may retain stale data across requests if signals aren't firing.

4.5 Dual Serialization: _raw Property

StructuredJSONField.contribute_to_class() adds a <field>_raw property to the Django model class. This returns the raw dict from from_db_value() without validation — useful for migrations and debugging.

4.6 Dynamic DRF Serializer Factory

build_model_serializer(model) creates a ModelSerializer subclass on-the-fly for any Django model. It maps StructuredJSONField to JSONFieldInnerSerializer and respects MAX_DEPTH to prevent infinite recursion.

4.7 PATCH Deep Merge

The DRF StructuredJSONField.to_internal_value() handles PATCH by deep-merging old + new data via dict_merge(), so partial updates don't wipe unmodified nested fields.

4.8 JSON Schema for Admin Widget

FK/QS fields add format: "select2" and type: "relation" to their JSON schema. The JSONEditor resolver detects this and uses the patched Select2 editor (patches.js), which makes AJAX calls to /structured_field/search_model/<model>/ for autocomplete.

4.9 Schema Migration Generator

The generate_schema_migration command:

1. Scans all DB rows for validation errors against the current schema.
2. Groups errors by type/location.
3. Generates a complete Django migration file with RunPython + transform functions.
4. Supports --analyze-only for inspection without generating files.

4.10 Python 3.14 / PEP 649 Support

BaseModelMeta._get_raw_annotations() handles deferred annotations using annotationlib.get_annotations(FORMAT.FORWARDREF) on Python 3.14+.

4.11 Extra = 'ignore' by Default

BaseModel uses ConfigDict(extra='ignore'), meaning unknown fields in JSON data are silently dropped. This makes schema evolution forward-compatible — no need to migrate data when removing fields.

5. Coding Conventions

Python

• Import style: Standard library → third-party → Django → local. Absolute imports within the structured package.
• No docstrings in most internal code: Code is self-documenting; inline comments used sparingly for non-obvious logic.
• Type annotations: Used extensively throughout. Pydantic models are fully typed. Internal utilities use basic type hints.
• Naming:
  • Classes: PascalCase
  • Functions/methods: snake_case
  • Constants: UPPER_SNAKE_CASE
  • Private methods: _single_underscore_prefix
  • File names: snake_case.py
• Note: StructuredDescriptior is the established spelling (typo preserved for backward compatibility). Do NOT rename it.
• Error mapping: Pydantic ValidationError is always converted to nested dicts via map_pydantic_errors() before surfacing to Django/DRF.
• Field registration in __init__: Custom Pydantic fields define their core schema via __get_pydantic_core_schema__ classmethod.
• Metaclass usage: BaseModelMeta extends Pydantic's ModelMetaclass and must be maintained carefully — annotation patching order matters.
• Settings access: Always through the Settings singleton (structured.settings), never directly from django.conf.settings.

Frontend (JavaScript / SCSS)

• Plain ES modules: No framework (React/Vue). Uses vanilla JS + JSONEditor library.
• IIFE output: Rollup bundles to IIFE format for Django admin (no module loader).
• Class extension: patches.js extends JSONEditor.defaults.editors.select2 using ES6 class syntax.
• Django CSS variables: SCSS uses Django admin CSS custom properties (--body-bg, --border-color, --error-fg, etc.) for theme integration.
• Data flow: JSON schema + form data passed via HTML data-* attributes on the widget <div>. Editor changes write back to a hidden <textarea>.
• Build command: pnpm run build (or npx rollup -c) — outputs to structured/static/.

Tests

• Framework: pytest + pytest-django.
• Markers: All DB tests use @pytest.mark.django_db.
• Cache parametrization: Most tests are decorated with @pytest.mark.parametrize("cache_setting_fixture", ["cache_enabled", "cache_disabled", "shared_cache"], indirect=True) to run across all cache configurations.
• Late imports: Models are imported inside test functions, not at module top level.
• Standalone functions: Tests are top-level pytest functions (not classes), except TestRestFramework which uses a class with setup_method.
• Query counting: django_assert_num_queries from pytest-django for cache/performance assertions.
• Admin tests: Use admin_client fixture, POST data as strings, check HTTP 302 redirects.
• DRF tests: Use rest_framework.test.APIClient.
• Settings override: conftest.py provides indirect parametrize fixtures that modify settings.STRUCTURED_FIELD dict.
• Test DB: SQLite at test_db.sqlite3, managed by session-scoped django_db_setup fixture.

6. How to Add or Modify Features

Adding a New Pydantic Field Type

1. Create a new file in structured/pydantic/fields/ (e.g., mytype.py).
2. Define a class with __class_getitem__ (for generic syntax) and __get_pydantic_core_schema__.
3. Implement validators (from raw data) and serializer (to JSON).
4. If it references Django models, add relationship handling in structured/cache/engine.py — register a new RelInfo type.
5. If it needs admin widget support, add JSON schema metadata (e.g., format, type keys) and handle it in reactive-forms/patches.js.
6. Export from structured/pydantic/fields/__init__.py.
7. Add annotation patching support in structured/utils/pydantic.py → patch_annotation() if auto-conversion is desired.

Adding a New Utility

1. Create a file in structured/utils/.
2. Use pure functions with type hints.
3. Import into structured/utils/__init__.py if it should be part of the public API.

Modifying the Cache System

• Cache population: structured/cache/engine.py → _populate_cache().
• Cache invalidation: structured/cache/cache.py → signal handlers.
• Relationship discovery: engine.py → _get_rel_info() and get_all_fk_data().
• Shared cache singleton: cache.py → ThreadSafeCache.
• Important: The cache wraps raw PK values with ValueWithCache before Pydantic validation, then resolves them after. This two-phase design is critical.

Modifying the Admin Widget

• Python side: structured/widget/widgets.py manages which CSS/JS files are loaded, and structured/widget/fields.py handles form validation.
• Template: structured/templates/json-forms/widget.html — the data-schema, data-formdata, data-uischema attributes.
• JS side: reactive-forms/main.js initializes JSONEditor, callbacks.js configures AJAX, patches.js extends Select2 for relations.
• After JS/SCSS changes, rebuild: pnpm run build (or npx rollup -c rollup.config.mjs).
• Built assets go to structured/static/js/ and structured/static/css/.

Modifying DRF Integration

• structured/contrib/restframework.py contains the serializer field, mixin, and StructuredModelSerializer.
• to_representation uses model_dump() with context depth tracking.
• to_internal_value handles PATCH via dict_merge and validates via Pydantic schema.
• Error mapping uses map_pydantic_errors().

Adding/Modifying Migrations Support

• structured/migrations/structured_json_migration.py — the generator class.
• structured/management/commands/generate_schema_migration.py — the CLI entry point.
• The generator analyzes live data via analyze_validation_errors() and produces Python code strings.

7. How to Write Tests

Test Location & Naming

• Test files go in tests/ with test_ prefix.
• Use standalone functions (not classes) unless shared setup is needed.

Required Patterns

import pytest

@pytest.mark.django_db
@pytest.mark.parametrize(
    "cache_setting_fixture",
    ["cache_enabled", "cache_disabled", "shared_cache"],
    indirect=True,
)
def test_my_feature(cache_setting_fixture):
    # Import models inside the test function
    from tests.app.test_module.models import TestModel, SimpleRelationModel

    # Create test data
    rel = SimpleRelationModel.objects.create(name="Test")
    obj = TestModel.objects.create(
        title="Test",
        structured_data={"name": "test", "age": 25, "fk_field": rel.pk},
    )

    # Refresh and assert
    obj.refresh_from_db()
    assert obj.structured_data.name == "test"
    assert obj.structured_data.fk_field == rel

Query Count Tests

@pytest.mark.django_db
def test_query_efficiency(django_assert_num_queries):
    from tests.app.test_module.models import TestModel
    obj = TestModel.objects.create(...)
    with django_assert_num_queries(N):
        obj.refresh_from_db()
        _ = obj.structured_data.some_field

DRF Tests

from rest_framework.test import APIClient

class TestMyAPI:
    def setup_method(self):
        self.client = APIClient()
        # Create test data

    @pytest.mark.django_db
    def test_get(self):
        response = self.client.get("/api/testmodels/1/")
        assert response.status_code == 200

Test Models

All test schemas and Django models live in tests/app/test_module/models.py. If a new model/schema is needed for testing, add it there and run python manage.py makemigrations test_module.

Running Tests

# All tests
pytest -v

# Specific test file
pytest tests/test_structured_field.py -v

# With coverage
pytest --cov=structured -s -vv --cov-report=xml --cov-report=term-missing

# Via Makefile
make test

8. Common Debugging Scenarios

"N+1 queries" / Performance Issues

• Check if STRUCTURED_FIELD['CACHE']['ENABLED'] is True.
• Inspect CacheEngine.get_all_fk_data() — verify all FK/QS fields are discovered.
• Use django_assert_num_queries in tests to verify.
• Look at _get_rel_info() in engine.py to confirm relationship type detection.

Pydantic Validation Errors

• StructuredJSONField.validate() calls schema.validate_python(cast_to_python(value)).
• Errors are mapped via map_pydantic_errors() → nested dict keyed by field path.
• For DRF, errors surface as DRF ValidationError with the same nested structure.
• Check patch_annotation() in utils/pydantic.py if annotation rewriting is suspected.

Admin Widget Not Rendering

• Ensure structured is in INSTALLED_APPS.
• Ensure path("", include("structured.urls")) is in URL config.
• Check that formfield() in fields.py returns StructuredJSONFormField with correct schema.
• Verify static files are built: structured/static/js/structured-field-form.min.js must exist.

Select2 Autocomplete Not Working

• The AJAX endpoint is /structured_field/search_model/<app_label>.<model_name>/.
• views.py → search() requires staff authentication.
• Check build_relation_schema_options() in utils/options.py for JSON schema metadata.
• Frontend: callbacks.js sends _q (search term) and page params.
• patches.js resolver checks schema.type === "relation" and schema.format === "select2".

Forward Reference Resolution Issues

• BaseModelMeta builds a namespace from module globals + parent frame.
• utils/namespace.py → merge_cls_and_parent_ns().
• Python 3.14: Uses annotationlib.get_annotations(FORMAT.FORWARDREF).
• Check LazyType in utils/typing.py for dot-path type resolution.

Cache Invalidation Problems

• Cache uses post_save / pre_delete signals.
• ThreadSafeCache (CACHE.SHARED = True) is a singleton — may retain stale data across requests if signals aren't firing.
• flush(filters=None) clears cache optionally per-model.
• Inspect signal connections in cache.py.

Schema Migration Issues

• Run python manage.py generate_schema_migration <app> <model> --analyze-only first.
• Generated migration is a template — humans must review and customize transform functions.
• extra='ignore' means removed fields don't need migration (silently dropped).
• New required fields without defaults DO need data migration.

9. Settings Reference

Configure via STRUCTURED_FIELD dict in Django settings:

STRUCTURED_FIELD = {
    "CACHE": {
        "ENABLED": True,    # Enable FK/QS cache (default: True)
        "SHARED": False,    # Use global thread-safe cache (default: False, experimental)
    },
    "SERIALIZATION": {
        "MAX_DEPTH": 2,     # Max depth for nested model serialization (default: 2)
    },
}

Access in code: from structured.settings import Settings; Settings().CACHE_ENABLED.

10. Build & Development Commands

# Python
pip install -e ".[dev]"            # Install in dev mode
pytest -v                          # Run tests
make test                          # Lint (flake8) + tests with coverage
python manage.py makemigrations    # After model changes
python manage.py migrate           # Apply migrations
python manage.py generate_schema_migration <app> <model>  # Schema migration

# Frontend
pnpm install                       # Install JS dependencies
pnpm run build                     # Build JS/CSS → structured/static/
npx rollup -c rollup.config.mjs    # Manual build
# Rollup config: reactive-forms/main.js → structured/static/js/ + css/

# Versioning
# Managed by python-semantic-release via pyproject.toml
# Version stored in: setup.py (__version__) and structured/__init__.py (__version__)

11. Important Warnings

1. Do NOT rename StructuredDescriptior — it's a known typo preserved for backward compatibility.
2. Annotation patching order matters in BaseModelMeta.__new__() — patch_annotation() must run before super().__new__() calls Pydantic's metaclass.
3. Cache two-phase design: ValueWithCache wrappers are injected before Pydantic validation and resolved after. Do not change this order.
4. extra='ignore' is intentional on BaseModel — do not change to 'forbid' without considering all downstream migration implications.
5. Models imported inside tests — this is intentional to avoid import-time side effects with Django app registry.
6. Test DB (test_db.sqlite3) is created by session-scoped fixture — do not commit it or rely on its state between test runs programmatically.
7. Frontend build output (structured/static/) is committed to the repo — after JS/SCSS changes, rebuild before committing.
8. Abstract model FK resolution uses a model key in the dict data to determine the concrete model class. QuerySet fields do NOT support abstract models.

Source: bnznamco/django-structured-field — distributed by TomeVault.