Build Data Explorer App

Build Data Explorer App

Overview

Build a full-stack Data Explorer App for Weaviate Collections with FastAPI.

Read first:

• Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
• Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters

Instructions

Core Rules

• Use a virtual environment via venv
• Use uv for Python project/dependency management.
• Do not manually author pyproject.toml or uv.lock; let uv generate/update them.
• Use this backend install set:
  • uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv
• Depending on user request: consider combining this app with the Query Agent Chatbot.
  • If the user explicitly only wants a data viewer/explorer, create this app independently
  • If the user wants a fully featured chat and data explorer, combine the apps
  • If no explicit instructions are given, ask the user their preference before continuing
  • See the Next Steps section for more details

Fast Setup Commands

Project bootstrap:

uv init data_explorer
cd data_explorer
uv venv
uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv

Workflow Contract

1. Build backend and frontend in one pass.
2. Create .env from the canonical template in environment_requirements.md, then add app-specific fields (for example, CORS_ORIGINS).
3. Before asking user to fill env, do non-secret local sanity checks that do not require real credentials (imports/compile/startup-shape checks).
4. Ask user to fill real env values:
   • Mandatory: WEAVIATE_URL, WEAVIATE_API_KEY
   • Optional: only provider keys required by their collection setup
5. After the user confirms, verify backend starts without errors and provide exact commands to run in the terminal.

Do not ask avoidable questions that you can resolve from context.

Directory Structure

Use a modular layout like:

data_explorer/
  backend/
    app/
      main.py
      config.py
      lifespan.py
      dependencies.py
      routers/
      services/
      models/
    .env  # local file, never committed

Keep these boundaries:

• routers: HTTP only
• services: business/query-agent logic
• models: request/response schemas
• config/lifespan: wiring and startup/shutdown

Backend Requirements

• FastAPI async app with lifespan.
• Async Weaviate client initialized in lifespan and closed on shutdown.
• Ensure no async blocking operations.
• Not a full CRUD implementation - this is only for viewing data in a Weaviate collection.
• Endpoints for:
  • GET /health
  • GET /env_check: returns what API keys are missing (if any) for verification on app start
  • GET /collections: return available collections
  • GET /data/{collection_name}?xx=xx&yy=yy: return data with optional arguments (more later), and pagination
• Pydantic settings should read from process environment; local .env loading is optional for local development.
• Conversation history mapping to Weaviate chat message format.

Env Rules

Mandatory:

• WEAVIATE_URL
• WEAVIATE_API_KEY

External provider keys:

• Include every provider key needed by the target collections.
• Leave unused provider keys empty/commented.

CORS:

• Default CORS_ORIGINS should include:
  • http://localhost:3000
  • http://127.0.0.1:3000
  • http://localhost:5173
  • http://127.0.0.1:5173

FastAPI standards

1. Do not use hardcoded status values, use status from FastAPI, for example:

from fastapi import status
status.HTTP_200_OK # code 200
status.HTTP_404_NOT_FOUND # code 404
# and more

2. Use a Pydantic BaseModel for the request and response_model in all endpoints that require it. Ensure schema validation to mitigate user-error on the API.

3. Use path parameters and query parameters for GET endpoints instead of payloads, for example:

@app.get("/items/{item_id}")
async def read_item(item_id: str):
    return {"item_id": item_id}

@app.get("/items/")
async def read_item(skip: int = 0, limit: int = 10):
    return fake_items_db[skip : skip + limit]

4. Implement best practices for error-handling, do early returns and provide the correct status codes when necessary.

5. Use proper logging for API usage, not simple print statements.

FastAPI endpoints

Basic structure of endpoints. Customise according to user preference or suitability. Do not follow exactly, this is a guideline only.

Ensure you also set up standard FastAPI procedures, such as global error handling, logging, dependencies. Set up an async client manager that connects on startup (via lifespan) and closes gracefully on app exit, use a dependency injection to add the client to the relevant endpoints.

GET /health

This is a standard health check. For example:

from pydantic import BaseModel

class HealthResponse(BaseModel):
    status: str

@app.get("/health", tags=["health"], response_model=HealthResponse)
async def health_check() -> HealthResponse:
    logger.info("Health check requested")
    return HealthResponse(status="healthy")

GET /env_check

Check what environment variables the backend has access to, used to verify the user's Weaviate configuration is correct. For example:

import os
from pydantic import BaseModel

class EnvCheckResponse(BaseModel):
    weaviate_url: bool
    weaviate_api_key: bool

@app.get("/env_check", tags=["health"])
async def env_check() -> EnvCheckResponse:
    logger.info("Environment check requested")
    return EnvCheckResponse(
        weaviate_url = os.getenv("WEAVIATE_URL") is not None,
        weaviate_api_key = os.getenv("WEAVIATE_API_KEY") is not None,
    )

GET /collections

Check what collections are available. For example:

from pydantic import BaseModel
from weaviate.client import WeaviateAsyncClient

class CollectionsResponse(BaseModel):
    collections: list[str]

@app.get("/collections", tags=["collections"])
async def collections() -> CollectionsResponse:

    # include client management to import async client here

    logger.info("Collections requested")
    collections = await client.collections.list_all()
    return CollectionsResponse(
        collections = list(collections.keys())
    )

Tip: consider expanding this endpoint to include collection descriptions and configs. await client.collections.list_all() returns dict[str, _CollectionConfigSimple] where _CollectionConfigSimple contains attributes:

• description: str
• properties: list[Property] where Property has .name, .description and .data_type (accessed via .data_type[:] to get name of data type as string)
• vector_config: dict[str, _NamedVectorConfig] where _NamedVectorConfig has attribute .vectorizer.vectorizer (not a typo) which can be accessed via .vectorizer.vectorizer[:] to get the name of the vectoriser as a string.

Multi-tenancy should be checked via

config = await collection.config.get()
config.multi_tenancy_config.enabled # bool

This is not available in the _CollectionConfigSimple, it must be fetched from collection.config.get().

GET /data/{collection_name}

Retrieve data from a collection, using pagination, sorting and filters.

from weaviate.collections import CollectionAsync
from fastapi import Query
from pydantic import BaseModel
from typing import Any

async def get_collection_data_types(collection: CollectionAsync) -> dict[str, str]:
    config = await collection.config.get()
    properties = config.properties
    return {prop.name: prop.data_type[:] for prop in properties}

class GetDataResponse(BaseModel):
    data_types: dict[str, str]
    items: list[dict[str, Any]]

@router.post("/data/{collection_name}")
async def get_data(
    collection_name: str,
    page_size: int = Query(default=10, ge=1, le=100),
    page_number: int = Query(default=1, ge=1),
    query: str = Query(default=""),
    sort_on: str = Query(default=None),
    ascending: bool = Query(default=True),
) -> GetDataResponse:

    # include client management to import async client here

    collection = await client.collections.use(collection_name)
    data_types = await async_get_collection_data_types(collection)

    if query != "":
        response = await collection.query.bm25(
            query=query,
            limit=page_size,
            offset=page_size * (page_number - 1),
        )
    elif sort_on is not None:
        response = await collection.query.fetch_objects(
            sort=Sort.by_property(name=sort_on, ascending=ascending),
            limit=page_size,
            offset=page_size * (page_number - 1),
        )
    else:
        response = await collection.query.fetch_objects(
            limit=page_size,
            offset=page_size * (page_number - 1),
        )

    return GetDataResponse(data_types = data_types, items = [obj.properties for obj in response.objects])

Tip: some collections can have multi-tenancy. Consider adding the tenant as an optional query parameter to get_data, e.g.

async def get_data(
    ... # existing args
    tenant: str | None = Query(default=None)
):
    base_collection = await client.collections.use(collection_name)
    data_types = await async_get_collection_data_types(collection)

    config = await collection.config.get()
    if config.multi_tenancy_config.enabled and tenant and tenant.strip():
        collection = base_collection.with_tenant(tenant)
    else:
        collection = base_collection

    # ...existing code

Post-Env Hand-Holding (Required)

After user says required env values are set, provide the terminal commands to run the backend:

cd data_explorer/backend
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

Then:

• Ask user to start terminal.
• Run smoke tests yourself against running services.
• Report pass/fail in plain language and fix blockers.

Do not offload detailed testing steps to the user unless they explicitly ask.

Troubleshooting

• Weaviate startup host errors: ensure WEAVIATE_URL is full https://... URL.
• For any other issues, refer to the official library/package documentation using web search.

Done Criteria

• Backend healthy.
• All endpoints work.
• User can run server in terminal with provided commands.

Next Steps

This application is currently a data explorer backend. You may optionally offer to integrate it with the Query Agent Chatbot based on user preference.

If the user chooses to combine these two applications, implement the integration as follows:

• Create or use a directory /routes which separate functions for query agent chat and data exploration. Import the routers in the main.py file
• If a frontend is requested, the frontend should have multiple pages/tabs depending on design choices so that data exploration and chat is separated
• Consider crossovers between functionalities, e.g. a chat button from the data viewer/collection viewer which takes the user to chat with that collection selected.
• Run quick tests to ensure the integration is seamless and the user can use both the chatbot and data explorer without any issues.

Frontend

When the user explicitly asks for a frontend, use this reference as guideline:

• Frontend Interface: Build a Next.js frontend to interact with the Weaviate backend.