Hologres Knowledge Base

Hologres Knowledge Base Management

Build enterprise search & RAG knowledge bases on Hologres in three steps: Create Table (with indexes) → Ingest Documents (with embeddings) → Search & Q&A

Unlike fully managed RAG services (e.g. ADBPG Vector Engine), Hologres exposes the underlying building blocks (vector + full-text + scalar columns in one table) and gives full control via SQL or the Python holo-search-sdk. This is the right choice when you need:

• A single table that can be searched by vector, keyword, and SQL filters in one query
• Tight integration with your existing OLAP / data warehouse on Hologres
• Custom chunking / embedding pipelines

Architecture: Hologres Instance + Schema + Table (vector column + text column + scalar columns) + HGraph Index + Fulltext Index + Embedding Model

Core Concepts

Version Requirements

[MUST] Verify version first:

hologres sql run "SELECT version()"

If < V4.0, full-text and HGraph indexes are unavailable — abort and notify the user.

Environment Setup

1. Install dependencies

# Hologres CLI (used for DDL, profile management, ai/model commands)
pip install hologres-cli

# holo-search-sdk (used for ingestion and search at runtime, requires Python >= 3.8)
pip install --upgrade holo-search-sdk    # >= 0.3.0
pip install psycopg-binary               # required by holo-search-sdk

2. Configure profile (one-time)

hologres config         # interactive wizard
hologres status         # verify connection

3. Set skill tag for query attribution

export HOLOGRES_SKILL=hologres-knowledge-base

All queries from this session will appear in hologres.hg_query_log with application_name = "hologres-cli/hologres-knowledge-base".

Parameter Confirmation

IMPORTANT: Parameter Confirmation — Before creating tables / indexes / inserting data, ALL user-customizable parameters MUST be confirmed with the user. Do not assume defaults.

Security: Never echo / print AccessKey, password, or API keys. Use hologres config and hologres model create --api-key <key> to store secrets outside the conversation. The CLI redacts sensitive literals in ~/.hologres/sql-history.jsonl.

Core Workflow

1. Create the Knowledge Base Table

A knowledge base in Hologres is a single column-store table holding:

• A primary key (id)
• The raw text chunk (content TEXT) — full-text indexed
• The embedding (embedding FLOAT4[]) — HGraph vector indexed
• Optional metadata columns (doc_id, chunk_idx, source, publish_date, …)

Option A — Recommended: one CREATE TABLE with all properties in WITH (V2.1+)

This is the official Hologres-recommended pattern: table columns, primary key, all table properties (orientation, distribution_key, clustering_key, event_time_column, …) and the vectors HGraph config all in a single WITH (...) clause. Then add the fulltext index separately (it is its own DDL).

⚠️ Avoid the legacy CALL set_table_property(...) form — it still works but the WITH syntax is more concise, atomic, and recommended from Hologres V2.1 onward. The bundled hologres table create CLI command currently emits the legacy syntax and does not support vectors=; use raw SQL via hologres sql run --write for this skill.

hologres sql run --write "
CREATE TABLE public.kb_documents (
    id BIGINT NOT NULL,
    doc_id TEXT NOT NULL,
    chunk_idx INT,
    content TEXT,
    embedding FLOAT4[] CHECK (array_ndims(embedding) = 1 AND array_length(embedding, 1) = 1024),
    source TEXT,
    publish_date TIMESTAMPTZ,
    PRIMARY KEY (doc_id, id)               -- composite PK so distribution_key ⊆ PK
)
WITH (
    orientation = 'column',
    distribution_key = 'doc_id',           -- co-locate chunks of the same doc on one shard
    clustering_key = 'publish_date:asc',
    event_time_column = 'publish_date',    -- file-level pruning on time range queries
    vectors = '{
        \"embedding\": {
            \"algorithm\": \"HGraph\",
            \"distance_method\": \"Cosine\",
            \"builder_params\": {
                \"base_quantization_type\": \"rabitq\",
                \"graph_storage_type\": \"compressed\",
                \"max_degree\": 64,
                \"ef_construction\": 400,
                \"use_reorder\": true,
                \"precise_quantization_type\": \"fp32\",
                \"extra_columns\": \"id\"
            }
        }
    }'
);

CREATE INDEX idx_kb_content_ft
    ON public.kb_documents
    USING FULLTEXT (content)
    WITH (tokenizer = 'jieba');
"

⚠️ precise_quantization_type is mandatory at CREATE TABLE time when use_reorder = true, even though the docs say it has a default of fp32. ALTER TABLE SET (vectors = ...) is more lenient and will apply the default — but CREATE TABLE WITH (vectors = ...) will reject the statement with ERROR: precise_quantization_type must be set here.

Option B — All-in-one via holo-search-sdk (Python)

The SDK exposes convenience methods (set_vector_index, create_text_index) that internally issue ALTER TABLE. You can still fold all base-table properties into the initial CREATE TABLE WITH clause — just skip the vectors config there and let set_vector_index() add it later (easier to call from Python than building a JSON literal in a multi-line string).

import holo_search_sdk as holo

client = holo.connect(
    host="<HOLO_HOST>", port=<HOLO_PORT>, database="<HOLO_DBNAME>",
    access_key_id="<ACCESS_KEY_ID>", access_key_secret="<ACCESS_KEY_SECRET>",
    schema="public",
)
client.connect()

client.execute("""
CREATE TABLE IF NOT EXISTS kb_documents (
    id BIGINT NOT NULL,
    doc_id TEXT NOT NULL,
    chunk_idx INT,
    content TEXT,
    embedding FLOAT4[] CHECK (array_ndims(embedding) = 1 AND array_length(embedding, 1) = 1024),
    source TEXT,
    publish_date TIMESTAMPTZ,
    PRIMARY KEY (doc_id, id)
)
WITH (
    orientation = 'column',
    distribution_key = 'doc_id',
    clustering_key = 'publish_date:asc',
    event_time_column = 'publish_date'
);
""", fetch_result=False)

table = client.open_table("kb_documents")

# Vector index (SDK issues ALTER TABLE under the hood — accepts default precise_quantization_type)
table.set_vector_index(
    column="embedding",
    distance_method="Cosine",
    base_quantization_type="rabitq",
    use_reorder=True,
    max_degree=64,
    ef_construction=400,
)

# Full-text index
table.create_text_index(
    index_name="idx_kb_content_ft",
    column="content",
    tokenizer="jieba",
)

Index design checklist

2. Ingest Documents

The ingestion pipeline is app-side: chunk → embed → INSERT. Hologres does not auto-chunk or auto-embed (unlike ADBPG). You have two embedding strategies:

Strategy A — Embed in Hologres via ai_gen() (recommended for SQL-only pipelines)

Register an external embedding model once:

hologres model create \
  --name my_embed \
  --type text-embedding-v4 \
  --api-key '<DASHSCOPE_API_KEY>'

Then insert chunks and let Hologres compute embeddings server-side:

hologres sql run --write "
INSERT INTO public.kb_documents (id, doc_id, chunk_idx, content, embedding, source, publish_date)
SELECT
    nextval('kb_documents_id_seq'),
    'doc_001',
    chunk_idx,
    chunk_text,
    ai_gen('my_embed', chunk_text)::FLOAT4[],
    'manual.pdf',
    NOW()
FROM (VALUES
    (1, 'Hologres 是一款实时数仓，支持向量检索…'),
    (2, '通过 HGraph 算法可实现高性能近似最近邻搜索…')
) AS t(chunk_idx, chunk_text);
"

Strategy B — Embed in Python, upsert via holo-search-sdk

For external embedding APIs or local models, generate embeddings client-side then bulk-upsert:

# Pseudo-code — replace embed() with your model call
chunks = chunk_document(text, chunk_size=500, chunk_overlap=50)

rows = []
for i, chunk in enumerate(chunks):
    embedding = embed(chunk)              # → list[float] of length vector_dim
    rows.append([next_id(), "doc_001", i, chunk, embedding, "manual.pdf", "2026-01-01"])

table.upsert_multi(
    index_column="id",
    values=rows,
    column_names=["id", "doc_id", "chunk_idx", "content", "embedding", "source", "publish_date"],
    update=True,
)

After bulk load — trigger index build

Both vector and fulltext indexes build asynchronously during Compaction. Until built, BM25 scores are 0 and HGraph search falls back to brute force. For batch imports:

# Preferred: use Serverless Computing (does Compaction + index build during load)
hologres sql run --write "SET hg_computing_resource = 'serverless';
SET hg_serverless_computing_run_compaction_before_commit_bulk_load = on;"

# Or manually trigger Compaction after load
hologres sql run --write "VACUUM public.kb_documents;"
# or, more aggressive:
hologres sql run --write "SELECT hologres.hg_full_compact_table('public.kb_documents', 'max_file_size_mb=4096');"

Verify index build progress

# Full-text index progress
hologres sql run "SELECT * FROM hg_show_build_index_progress('public.kb_documents');"

# Inspect existing indexes
hologres sql run "SELECT * FROM hologres.hg_index_properties WHERE table_name = 'kb_documents';"
hologres sql run "SELECT * FROM hologres.hg_table_properties WHERE table_name = 'kb_documents' AND property_key = 'vectors';"

3. Search & Q&A

Vector search (semantic similarity)

# holo-search-sdk
query_vec = embed("如何配置 Hologres 向量索引？")

results = (
    table.search_vector(
        vector=query_vec,
        column="embedding",
        distance_method="Cosine",
        output_name="similarity_score",
    )
    .where("publish_date > '2024-01-01'")     # scalar filter
    .order_by("similarity_score", "desc")      # Cosine in Hologres returns similarity (higher = closer)
    .limit(10)
    .fetchall()
)

Equivalent raw SQL (verify via EXPLAIN — look for Vector Filter in the plan):

SELECT id, content,
       approx_cosine_distance(embedding, '<query_vector_literal>') AS distance
FROM public.kb_documents
WHERE publish_date > '2024-01-01'
ORDER BY distance DESC     -- Cosine in Hologres returns similarity-like score → DESC
LIMIT 10;

⚠️ Hologres "distance" functions for Cosine / InnerProduct return SIMILARITY scores (higher = more similar), not true distance. This is a naming quirk — always check the ORDER BY direction below.

distance_method	Use approx function	ORDER BY	Semantics
Euclidean	approx_euclidean_distance	ASC	True L2 distance (lower = closer)
InnerProduct	approx_inner_product_distance	DESC	Inner product (higher = closer)
Cosine	approx_cosine_distance	DESC	Similarity (higher = closer) — not 1 - cos

⚠️ The exact functions (cosine_distance, inner_product_distance) follow the SAME convention as their approx counterparts — they return similarity for Cosine/InnerProduct, not true distance. Only euclidean_distance returns a true distance.

⚠️ Distance function must match index distance_method, otherwise the HGraph index is bypassed and the query falls back to brute force (still correct, but slower).

Full-text search (keyword / BM25)

results = (
    table.search_text(
        column="content",
        expression="向量索引 配置",
        mode="match", operator="AND",        # require all terms
        return_score=True, return_score_name="bm25",
    )
    .order_by("bm25", "desc")
    .limit(10)
    .fetchall()
)

Equivalent raw SQL — Hologres provides a dedicated TEXT_SEARCH() function that returns BM25 score. Do not use PostgreSQL's @@ to_tsquery() — that targets the built-in tsvector, not the Tantivy-backed USING FULLTEXT index, and silently bypasses it.

-- Basic keyword match (default operator=OR)
SELECT id, content,
       text_search(content, '向量索引 配置') AS bm25
FROM public.kb_documents
WHERE text_search(content, '向量索引 配置') > 0     -- > 0 = matched
ORDER BY bm25 DESC
LIMIT 10;

-- AND mode: require all tokens
SELECT id, content,
       text_search(content, '向量索引 配置', operator => 'AND') AS bm25
FROM public.kb_documents
WHERE text_search(content, '向量索引 配置', operator => 'AND') > 0
ORDER BY bm25 DESC;

Verify the index is used via EXPLAIN — look for Fulltext Filter in the plan.

Search modes (mode => ...):

phrase mode slop unit: characters for jieba/keyword/icu; tokens for standard/simple/whitespace.

Debug tokenization: SELECT tokenize('Hologres V4.0 向量索引'); → {hologres,v4.0,v,4.0,向量,索引}

Hybrid search (vector + full-text + scalar)

Hologres lets you combine all three in a single SQL — this is its strongest differentiator vs. dedicated vector DBs. Below is RRF (Reciprocal Rank Fusion) — a simple, parameter-light fusion:

WITH vec AS (
  SELECT id, content,
         approx_cosine_distance(embedding, '<query_vector_literal>') AS vec_sim,
         ROW_NUMBER() OVER (
           ORDER BY approx_cosine_distance(embedding, '<query_vector_literal>') DESC
         ) AS vec_rank
  FROM public.kb_documents
  WHERE publish_date > '2024-01-01'
  ORDER BY vec_sim DESC LIMIT 50
),
ft AS (
  SELECT id, content,
         text_search(content, 'Hologres 向量索引') AS bm25,
         ROW_NUMBER() OVER (
           ORDER BY text_search(content, 'Hologres 向量索引') DESC
         ) AS ft_rank
  FROM public.kb_documents
  WHERE text_search(content, 'Hologres 向量索引') > 0
    AND publish_date > '2024-01-01'
  LIMIT 50
)
-- RRF merge (k=60 is the conventional smoothing constant)
SELECT
  COALESCE(vec.id, ft.id) AS id,
  COALESCE(vec.content, ft.content) AS content,
  COALESCE(1.0 / (60 + vec.vec_rank), 0)
  + COALESCE(1.0 / (60 + ft.ft_rank), 0) AS rrf_score
FROM vec FULL OUTER JOIN ft USING (id)
ORDER BY rrf_score DESC
LIMIT 10;

Q&A (RAG)

Hologres has no built-in chat-with-knowledge-base API. The pattern is assemble prompt externally, then call any LLM — either via hologres ai gen (uses a model registered in Hologres) or via an external SDK:

# 1. Retrieve top-K chunks (use the hybrid query above)
# 2. Concatenate chunks into context, then ask the LLM:
hologres ai gen \
  "根据下面知识库片段回答问题。

【知识库】
$(cat retrieved_chunks.txt)

【问题】
如何为 Hologres 表添加 HGraph 向量索引？" \
  --model qwen-max

Index Maintenance

Modify an index

# Switch full-text tokenizer
hologres sql run --write "ALTER INDEX idx_kb_content_ft SET (tokenizer = 'ik');"

# Tune HGraph parameters (rebuilds on next Compaction)
hologres sql run --write "
ALTER TABLE public.kb_documents
SET (vectors = '{\"embedding\": {\"algorithm\": \"HGraph\", \"distance_method\": \"Cosine\",
                  \"builder_params\": {\"max_degree\": 96, \"ef_construction\": 600,
                  \"base_quantization_type\": \"rabitq\", \"use_reorder\": true}}}');
"
hologres sql run --write "VACUUM public.kb_documents;"   # trigger rebuild

Drop an index

hologres sql run --write "DROP INDEX idx_kb_content_ft;"

# Drop vector index (replace vectors map with empty JSON)
hologres sql run --write "ALTER TABLE public.kb_documents SET (vectors = '{}');"

Limitations & Gotchas

Reference Links

External:

• Hologres 全文倒排索引官方文档
• Hologres HGraph 向量索引官方文档
• Hologres holo-search-sdk 官方文档

Best Practices

1. Always trigger Compaction after bulk load — otherwise BM25 = 0 and vector search degrades. Prefer SET hg_computing_resource = 'serverless' for synchronous build during load.
2. Confirm vector dimension matches embedding model — dim mismatch is the #1 INSERT failure.
3. Use extra_columns on HGraph to fetch ID/scalar columns from the index — avoids reading the base table.
4. Pick rabitq + use_reorder=true as the default quantization — best precision/cost ratio.
5. For Chinese: jieba (general) or ik (terminology); for English: standard; for fuzzy: ngram.
6. Verify the index is used — EXPLAIN <query> and look for Vector Filter (vector) or fulltext-index scan.
7. Don't index everything — small (<100k rows) tables search faster by brute force.
8. Tag queries with HOLOGRES_SKILL — for per-skill stats in hg_query_log.

Error Codes (from hologres-cli)