project-knowledge

name	project-knowledge
description	LoongCollector project knowledge: architecture, terminology, codebase map, and coding standards (C++/Go).

LoongCollector Project Knowledge

Architecture Overview

The LoongCollector architecture is based on a plugin system with the following key components:

1. Core Application: Main entry point in core/logtail.cpp, initializes Application class in core/application/Application.cpp. Follows singleton pattern, manages overall lifecycle.

2. Plugin System: Supports plugins for data collection, processing, and flushing:

   • Inputs: Collect data from various sources (files, network, system metrics, etc.)
   • Processors: Transform and process collected data
   • Flushers: Send processed data to various backends

3. Pipeline Management: Collection pipelines managed by CollectionPipelineManager handle data flow from inputs through processors to flushers.

4. Configuration: Supports both local and remote configuration management with watchers that monitor for configuration changes.

5. Queuing System: Implements various queue types including bounded queues, circular queues, and exactly-once delivery queues for reliable data transmission.

6. Monitoring: Built-in monitoring and metrics collection for tracking the collector's own performance and health.

Project Structure

core/                    # Core C++ code
  plugin/                # Plugin system
    input/               # Data collection input plugins
    processor/           # Data processing plugins
    flusher/             # Data output plugins (SLS, file, etc.)
  collection_pipeline/   # Main pipeline flow (queue, batch, serialization)
  config/                # Configuration (loading, providers, feedback)
    provider/            # Config providers (Enterprise, Legacy)
  common/                # Common utilities, data structures, network, string, crypto
  monitor/               # Monitoring, metrics collection, alerting
  logger/                # Logging system
  checkpoint/            # Checkpoint, state management
  app_config/            # Global configuration
  models/                # Core data structures (events, logs, metrics)
  parser/                # Log parsers
  task_pipeline/         # Task scheduling
  go_pipeline/           # Go plugin integration
  ebpf/                  # eBPF collection and plugins
  host_monitor/          # Host-level monitoring
  shennong/              # Shennong metrics
  prometheus/            # Prometheus collection
  file_server/           # File collection and management
  container_manager/     # Container environment management
  application/           # Main application entry
  protobuf/              # Protobuf protocol definitions
  metadata/              # K8s and other metadata collection
  constants/             # Constants
  tools/                 # Internal utility scripts
  unittest/              # Unit tests
  legacy_test/           # Historical test cases

pkg/                     # Go packages
  helper/                # Go helper functions
    containercenter/     # Go container-related functions

plugin_main/             # Plugin main entry
pluginmanager/           # Go plugin manager (lifecycle, registration)
plugins/                 # Go plugin packages
  input/                 # Go input plugins (docker, etc.)
  processor/             # Go processor plugins
  flusher/               # Go flusher plugins
  aggregator/            # Go aggregator plugins
  extension/             # Go extension plugins
  all/                   # Plugin registration and init
  test/                  # Go plugin tests

test/                    # Integration tests
e2e/                     # E2E test cases (open source Go plugins)
e2e_enterprise/          # E2E enterprise test cases (host + K8s)
docs/                    # Project documentation
scripts/                 # Build, deploy, test scripts
docker/                  # Docker-related files
rpm/                     # RPM packaging
external/                # External dependencies