Core Application: Main entry point in core/logtail.cpp, initializes Application class in core/application/Application.cpp. Follows singleton pattern, manages overall lifecycle.
Plugin System: Supports plugins for data collection, processing, and flushing:
- Inputs: Collect data from various sources (files, network, system metrics, etc.)
- Processors: Transform and process collected data
- Flushers: Send processed data to various backends
Pipeline Management: Collection pipelines managed by CollectionPipelineManager handle data flow from inputs through processors to flushers.
Configuration: Supports both local and remote configuration management with watchers that monitor for configuration changes.
Queuing System: Implements various queue types including bounded queues, circular queues, and exactly-once delivery queues for reliable data transmission.
Monitoring: Built-in monitoring and metrics collection for tracking the collector's own performance and health.

Project Structure

core/                    # Core C++ code
  plugin/                # Plugin system
    input/               # Data collection input plugins
    processor/           # Data processing plugins
    flusher/             # Data output plugins (SLS, file, etc.)
  collection_pipeline/   # Main pipeline flow (queue, batch, serialization)
  config/                # Configuration (loading, providers, feedback)
    provider/            # Config providers (Enterprise, Legacy)
  common/                # Common utilities, data structures, network, string, crypto
  monitor/               # Monitoring, metrics collection, alerting
  logger/                # Logging system
  checkpoint/            # Checkpoint, state management
  app_config/            # Global configuration
  models/                # Core data structures (events, logs, metrics)
  parser/                # Log parsers
  task_pipeline/         # Task scheduling
  go_pipeline/           # Go plugin integration
  ebpf/                  # eBPF collection and plugins
  host_monitor/          # Host-level monitoring
  shennong/              # Shennong metrics
  prometheus/            # Prometheus collection
  file_server/           # File collection and management
  container_manager/     # Container environment management
  application/           # Main application entry
  protobuf/              # Protobuf protocol definitions
  metadata/              # K8s and other metadata collection
  constants/             # Constants
  tools/                 # Internal utility scripts
  unittest/              # Unit tests
  legacy_test/           # Historical test cases

pkg/                     # Go packages
  helper/                # Go helper functions
    containercenter/     # Go container-related functions

plugin_main/             # Plugin main entry
pluginmanager/           # Go plugin manager (lifecycle, registration)
plugins/                 # Go plugin packages
  input/                 # Go input plugins (docker, etc.)
  processor/             # Go processor plugins
  flusher/               # Go flusher plugins
  aggregator/            # Go aggregator plugins
  extension/             # Go extension plugins
  all/                   # Plugin registration and init
  test/                  # Go plugin tests

test/                    # Integration tests
e2e/                     # E2E test cases (open source Go plugins)
e2e_enterprise/          # E2E enterprise test cases (host + K8s)
docs/                    # Project documentation
scripts/                 # Build, deploy, test scripts
docker/                  # Docker-related files
rpm/                     # RPM packaging
external/                # External dependencies

Key Dependencies

Header-Only Libraries

spdlog - Logging
rapidjson - JSON parsing

Compiled Libraries

Testing: gtest, gmock
Serialization: protobuf
Regex: re2
Hash: cityhash
Config: jsoncpp, yamlcpp
Compression: lz4, zlib, zstd
Network: curl, ssl, crypto
System: boost, gflags, leveldb, uuid
Memory: tcmalloc (optional)

Tech Stack

C++ (main implementation, C++17/20)
Protobuf (data serialization)
eBPF (kernel-level data collection)
Prometheus (metrics collection)
Go (plugin adaptation)
Shell/Python (build and test scripts)

Terminology Glossary

Term	Description
LoongCollector	The observability data collection agent (formerly iLogtail)
Pipeline	A data processing chain: Input -> Processor(s) -> Flusher
Plugin	A modular component that performs specific data operations
Input Plugin	Collects data from a source (file, network, metric, etc.)
Processor Plugin	Transforms data (parse, filter, enrich, etc.)
Flusher Plugin	Sends data to a destination (SLS, stdout, Prometheus, etc.)
Config	Collection configuration defining pipeline behavior
Checkpoint	Persistent state tracking for exactly-once delivery
Runner	Execution wrapper for a specific plugin instance
Queue	Data buffer between pipeline stages
Batch	Group of events processed/sent together
SLS	Alibaba Cloud Simple Log Service
eBPF	Extended Berkeley Packet Filter (kernel tracing)
SPL	Structured Processing Language

Codebase Map

Key Entry Points and Core Flows

Path	Purpose
`core/logtail.cpp`	Main entry point
`core/application/Application.cpp`	Application singleton, lifecycle management
`core/collection_pipeline/CollectionPipelineManager.cpp`	Pipeline lifecycle
`core/collection_pipeline/CollectionPipeline.cpp`	Pipeline execution
`core/runner/ProcessorRunner.cpp`	Processor execution
`core/runner/FlusherRunner.cpp`	Flusher execution
`core/config/watcher/PipelineConfigWatcher.cpp`	Config change detection
`core/file_server/FileServer.cpp`	File collection management
`core/file_server/checkpoint/CheckpointManagerV2.cpp`	Exactly-once checkpoint

Invariant Rules

Lifecycle: All plugins follow Init -> Start -> Stop -> Close lifecycle
Resource Release: Every thread/future/queue must be properly cleaned up on stop
Config: Environment variables are case-insensitive with default fallbacks
Queue: Bounded queue with backpressure; pop on disabled queue should not hang
Hot Reload: After config change, system must return to consistent "collect+process+send" state

Common Patterns

RAII for resource management
Smart pointers over raw pointers
Singleton pattern for managers (Application, AlarmManager, WriteMetrics)
Thread-safe queues with condition variables
Plugin registration via static initialization

C++ Coding Standards

Naming

PascalCase for class names, global functions, public methods
camelCase for variable names and private methods
SCREAMING_SNAKE_CASE for macros and constants
m prefix for member variables (e.g., mUserId)
k prefix for constants (e.g., kMaxSendBufferSize)

Modern C++

Prefer C++17/20 features (auto, range-based loops, smart pointers)
Use std::unique_ptr / std::shared_ptr for memory management
Prefer std::optional, std::variant, std::any for type-safe alternatives
Use constexpr and const for compile-time computations
Use std::string_view for read-only string operations

Error Handling

Use exceptions for error handling (std::runtime_error, std::invalid_argument)
RAII for resource management to avoid memory leaks
Validate inputs at function boundaries
Log errors using spdlog

Performance

Avoid unnecessary heap allocations; prefer stack-based objects
Use std::move for move semantics
Optimize loops with <algorithm> (e.g., std::sort, std::for_each)
Use std::array or std::vector over raw arrays

Security

Avoid C-style casts; use static_cast, dynamic_cast, reinterpret_cast
Enforce const-correctness
Avoid global variables; use singletons sparingly
Use enum class for strongly typed enumerations

Testing

Unit tests using Google Test (GTest) / Google Mock
Integration tests for system components

Go Coding Standards

Naming

PascalCase for exported types and functions
camelCase for unexported types and functions
snake_case for variables and constants
Package names use lowercase

Error Handling

Return errors explicitly, do not panic
Use fmt.Errorf with %w for error wrapping
Check errors at every call site

Concurrency

Use goroutines for concurrent operations
Use channels or sync primitives for communication
Avoid goroutine leaks; always provide exit paths

Testing

Use standard testing package
Table-driven tests for function coverage
Integration tests via E2E framework