Surrealdb Ffi Codec

SurrealDB FFI Codec

Overview

An efficient binary codec pattern for accessing SurrealDB Embedded from other languages (Go, Swift, Kotlin, etc.) via FFI.

Core principle: eliminate JSON and optimize with a two‑layer structure:

• FlatBuffers: fixed schema (error codes, status, operation types)
• MessagePack: dynamic schema (query results, parameters)

When to Use

• Embedding SurrealDB in Rust and accessing it from other languages via FFI
• Eliminating JSON parse/serialize overhead
• Handling dynamic query results while preserving type safety
• Requiring zero‑copy access through FlatBuffers

When NOT to Use

• HTTP/WebSocket client implementations (use official SurrealDB SDKs)
• Server-side API development without FFI requirements
• Simple single-language Rust projects (use surrealdb crate directly)
• MCP server implementations (this skill provides templates, not runtime)

Important: This skill provides implementation templates and guidelines only, not executable binaries or runtime libraries.

Examples

User request: "Create an FFI layer to access SurrealDB from Go"

→ Generate Rust library with:

• FlatBuffers schemas from resources/schemas/
• Codec implementation from resources/codec/
• C ABI wrapper from resources/ffi/
• Go client code following docs/integration_guide.md

User request: "Convert SurrealDB query results to MessagePack"

→ Implement converter using resources/codec/converter.rs.template with proper TypeHints handling.

Conversion Flow

See docs/design_principle.md for detailed architecture. Summary:

sql::Value → rmpv::Value → [u8] bytes (in FlatBuffers payload)

Type Mapping

See docs/conversion_table.md for the complete type conversion reference.

Key points:

• sql::Number::Decimal → f64 (precision loss beyond 15-17 digits)
• sql::Datetime → RFC3339 string (remove d'...' wrapper)
• sql::Thing → ID-only string (remove table prefix and brackets)

Instructions

Project Layout

Generate the following structure:

rust_lib/
├── Cargo.toml
├── schemas/
│   ├── request.fbs
│   └── response.fbs
└── src/
    ├── lib.rs
    ├── codec/
    │   ├── mod.rs
    │   ├── converter.rs
    │   ├── msgpack.rs
    │   └── generated/
    └── ffi/
        ├── mod.rs
        ├── wrapper.rs
        └── error_codes.rs

Implementation

1. FlatBuffers schemas - Use resources/schemas/*.fbs.template, replace {{NAMESPACE}}
2. Codec - Use resources/codec/*.rs.template for type conversion
3. FFI wrapper - Use resources/ffi/*.rs.template for C ABI exports

FlatBuffers Code Generation

flatc --rust -o src/codec/generated schemas/request.fbs schemas/response.fbs

Dependencies

[dependencies]
surrealdb = { version = "2.x", features = ["kv-mem"] }  # or kv-rocksdb
flatbuffers = "24.x"
rmp-serde = "1.x"
rmpv = "1.x"
chrono = { version = "0.4", features = ["serde"] }
thiserror = "2.x"
log = "0.4"

Guidelines

Using TypeHints

Convert string parameters from clients into native SurrealDB types:

// Formats: "field_name" or "datetime:field_name" or "record:table:field"
let hints = TypeHints::parse(&[
    "created_at".to_string(),
    "record:user:owner_id".to_string(),
]);
let surreal_value = to_surreal_value_with_type_hints(msgpack_value, &hints);

Decimal Precision Loss

sql::Number::Decimal is converted to f64, so precision beyond 15–17 digits is lost. If exact precision is required, treat it as a string.

Thing ID Cleanup

SurrealDB 2.x wraps Thing IDs in angle brackets. Remove them during conversion:

// "user:⟨uuid-here⟩" -> "uuid-here"
let clean_id = id.trim_matches(|c| c == '\u{27E8}' || c == '\u{27E9}');

Error Handling

• status_code = 0: success
• status_code = -1: Rust panic (fatal)
• status_code > 0: application error

All FFI functions must wrap logic with std::panic::catch_unwind to prevent panics from crossing the FFI boundary.

Async Runtime Management

FFI functions are synchronous but SurrealDB SDK is async. Use OnceLock for global runtime singleton. See docs/integration_guide.md for the complete pattern.

Applying RecordFieldHint

FlatBuffers record_fields: [RecordFieldHint] should be mapped into TypeHints.record_fields.

record_fields = [
  { field: "owner_id", table_name: "user" }
]

The above is equivalent to record:user:owner_id. Ensure client hints and Rust TypeHints are consistent.

FFI Input Safety

• If request_ptr is null or request_len == 0, return InvalidRequest immediately.
• FFI callers must always invoke free_response_buffer and avoid double‑free.

Error Payload Format

Use a structured MessagePack payload on error:

{ "code": i32, "message": String, "details": Option<String> }

Clients should ignore unknown fields for forward compatibility.