# Project Knowledge > LoongCollector Project Knowledge - Skill: `alibaba/project-knowledge` (Agent Skill) - Install (CLI): `npx skillmds@latest add alibaba/project-knowledge` - Raw SKILL.md: https://api.skillmd.com/api/skills/alibaba/project-knowledge/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: alibaba (https://skillmd.com/u/alibaba) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/alibaba/project-knowledge --- # LoongCollector Project Knowledge ## Architecture Overview The LoongCollector architecture is based on a plugin system with the following key components: 1. **Core Application**: Main entry point in `core/logtail.cpp`, initializes `Application` class in `core/application/Application.cpp`. Follows singleton pattern, manages overall lifecycle. 2. **Plugin System**: Supports plugins for data collection, processing, and flushing: - **Inputs**: Collect data from various sources (files, network, system metrics, etc.) - **Processors**: Transform and process collected data - **Flushers**: Send processed data to various backends 3. **Pipeline Management**: Collection pipelines managed by `CollectionPipelineManager` handle data flow from inputs through processors to flushers. 4. **Configuration**: Supports both local and remote configuration management with watchers that monitor for configuration changes. 5. **Queuing System**: Implements various queue types including bounded queues, circular queues, and exactly-once delivery queues for reliable data transmission. 6. **Monitoring**: Built-in monitoring and metrics collection for tracking the collector's own performance and health. ## Project Structure ``` core/ # Core C++ code plugin/ # Plugin system input/ # Data collection input plugins processor/ # Data processing plugins flusher/ # Data output plugins (SLS, file, etc.) collection_pipeline/ # Main pipeline flow (queue, batch, serialization) config/ # Configuration (loading, providers, feedback) provider/ # Config providers (Enterprise, Legacy) common/ # Common utilities, data structures, network, string, crypto monitor/ # Monitoring, metrics collection, alerting logger/ # Logging system checkpoint/ # Checkpoint, state management app_config/ # Global configuration models/ # Core data structures (events, logs, metrics) parser/ # Log parsers task_pipeline/ # Task scheduling go_pipeline/ # Go plugin integration ebpf/ # eBPF collection and plugins host_monitor/ # Host-level monitoring shennong/ # Shennong metrics prometheus/ # Prometheus collection file_server/ # File collection and management container_manager/ # Container environment management application/ # Main application entry protobuf/ # Protobuf protocol definitions metadata/ # K8s and other metadata collection constants/ # Constants tools/ # Internal utility scripts unittest/ # Unit tests legacy_test/ # Historical test cases pkg/ # Go packages helper/ # Go helper functions containercenter/ # Go container-related functions plugin_main/ # Plugin main entry pluginmanager/ # Go plugin manager (lifecycle, registration) plugins/ # Go plugin packages input/ # Go input plugins (docker, etc.) processor/ # Go processor plugins flusher/ # Go flusher plugins aggregator/ # Go aggregator plugins extension/ # Go extension plugins all/ # Plugin registration and init test/ # Go plugin tests test/ # Integration tests e2e/ # E2E test cases (open source Go plugins) e2e_enterprise/ # E2E enterprise test cases (host + K8s) docs/ # Project documentation scripts/ # Build, deploy, test scripts docker/ # Docker-related files rpm/ # RPM packaging external/ # External dependencies ``` ## Key Dependencies ### Header-Only Libraries - `spdlog` - Logging - `rapidjson` - JSON parsing ### Compiled Libraries - **Testing**: `gtest`, `gmock` - **Serialization**: `protobuf` - **Regex**: `re2` - **Hash**: `cityhash` - **Config**: `jsoncpp`, `yamlcpp` - **Compression**: `lz4`, `zlib`, `zstd` - **Network**: `curl`, `ssl`, `crypto` - **System**: `boost`, `gflags`, `leveldb`, `uuid` - **Memory**: `tcmalloc` (optional) ### Tech Stack - C++ (main implementation, C++17/20) - Protobuf (data serialization) - eBPF (kernel-level data collection) - Prometheus (metrics collection) - Go (plugin adaptation) - Shell/Python (build and test scripts) ## Terminology Glossary | Term | Description | |------|-------------| | LoongCollector | The observability data collection agent (formerly iLogtail) | | Pipeline | A data processing chain: Input -> Processor(s) -> Flusher | | Plugin | A modular component that performs specific data operations | | Input Plugin | Collects data from a source (file, network, metric, etc.) | | Processor Plugin | Transforms data (parse, filter, enrich, etc.) | | Flusher Plugin | Sends data to a destination (SLS, stdout, Prometheus, etc.) | | Config | Collection configuration defining pipeline behavior | | Checkpoint | Persistent state tracking for exactly-once delivery | | Runner | Execution wrapper for a specific plugin instance | | Queue | Data buffer between pipeline stages | | Batch | Group of events processed/sent together | | SLS | Alibaba Cloud Simple Log Service | | eBPF | Extended Berkeley Packet Filter (kernel tracing) | | SPL | Structured Processing Language | ## Codebase Map ### Key Entry Points and Core Flows | Path | Purpose | |------|---------| | `core/logtail.cpp` | Main entry point | | `core/application/Application.cpp` | Application singleton, lifecycle management | | `core/collection_pipeline/CollectionPipelineManager.cpp` | Pipeline lifecycle | | `core/collection_pipeline/CollectionPipeline.cpp` | Pipeline execution | | `core/runner/ProcessorRunner.cpp` | Processor execution | | `core/runner/FlusherRunner.cpp` | Flusher execution | | `core/config/watcher/PipelineConfigWatcher.cpp` | Config change detection | | `core/file_server/FileServer.cpp` | File collection management | | `core/file_server/checkpoint/CheckpointManagerV2.cpp` | Exactly-once checkpoint | ### Invariant Rules - **Lifecycle**: All plugins follow Init -> Start -> Stop -> Close lifecycle - **Resource Release**: Every thread/future/queue must be properly cleaned up on stop - **Config**: Environment variables are case-insensitive with default fallbacks - **Queue**: Bounded queue with backpressure; pop on disabled queue should not hang - **Hot Reload**: After config change, system must return to consistent "collect+process+send" state ### Common Patterns - RAII for resource management - Smart pointers over raw pointers - Singleton pattern for managers (Application, AlarmManager, WriteMetrics) - Thread-safe queues with condition variables - Plugin registration via static initialization ## C++ Coding Standards ### Naming - **PascalCase** for class names, global functions, public methods - **camelCase** for variable names and private methods - **SCREAMING_SNAKE_CASE** for macros and constants - **m** prefix for member variables (e.g., `mUserId`) - **k** prefix for constants (e.g., `kMaxSendBufferSize`) ### Modern C++ - Prefer C++17/20 features (auto, range-based loops, smart pointers) - Use `std::unique_ptr` / `std::shared_ptr` for memory management - Prefer `std::optional`, `std::variant`, `std::any` for type-safe alternatives - Use `constexpr` and `const` for compile-time computations - Use `std::string_view` for read-only string operations ### Error Handling - Use exceptions for error handling (`std::runtime_error`, `std::invalid_argument`) - RAII for resource management to avoid memory leaks - Validate inputs at function boundaries - Log errors using spdlog ### Performance - Avoid unnecessary heap allocations; prefer stack-based objects - Use `std::move` for move semantics - Optimize loops with `` (e.g., `std::sort`, `std::for_each`) - Use `std::array` or `std::vector` over raw arrays ### Security - Avoid C-style casts; use `static_cast`, `dynamic_cast`, `reinterpret_cast` - Enforce const-correctness - Avoid global variables; use singletons sparingly - Use `enum class` for strongly typed enumerations ### Testing - Unit tests using Google Test (GTest) / Google Mock - Integration tests for system components ## Go Coding Standards ### Naming - **PascalCase** for exported types and functions - **camelCase** for unexported types and functions - **snake_case** for variables and constants - Package names use lowercase ### Error Handling - Return errors explicitly, do not panic - Use `fmt.Errorf` with `%w` for error wrapping - Check errors at every call site ### Concurrency - Use goroutines for concurrent operations - Use channels or sync primitives for communication - Avoid goroutine leaks; always provide exit paths ### Testing - Use standard `testing` package - Table-driven tests for function coverage - Integration tests via E2E framework