Results for “data-ingestion”

Ingests music into a LanceDB via ncmdump-rs and sf-cli, searching and downloading from Netease Cloud Music or Bilibili, decrypting local NCM files, and extracting metadata, lyrics, and cover art.

Search archived video using natural language, ingest video files or RTSP streams, and manage ingested sources.

Ingests, normalizes, and enriches structured and unstructured threat intelligence feeds into STIX 2.1 format, evaluating feed quality and deduplicating indicators for distribution to SIEM, firewall, and EDR platforms.

Designs and implements big data architectures, processes large-scale datasets with distributed systems, and optimizes data pipelines for throughput using Hadoop, Spark, and cloud platforms.

Ingest, index, search, edit, and generate video and audio content from files, URLs, live streams, or desktop capture.

Upserts a validated MDX book corpus into Supabase via Drizzle, hydrating books, chapters, sections, and chunks tables while preserving stable bookmark anchors and only re-embedding changed content.

Processes STIX 2.1 threat intelligence bundles from TAXII 2.1 servers, normalizing objects into platform-native schemas and routing them to consuming systems.

Builds a configurable scraping agent that collects data from APIs, HTML, or RSS, enriches it with Gemini AI scoring, and stores results in Notion, Google Sheets, Supabase, or local files.