logging-ops

name	logging-ops
description	Query and analyze Kubernetes and VictoriaLogs data with kubectl, curl, and LogsQL. Use for Pod logs, error patterns, startup failures, timeouts, VictoriaLogs operations, fluent-bit or Vector collection issues, and trace correlation. 适用于日志查询、日志分析、VictoriaLogs 运维、错误排查和日志采集配置。

Log Operations Guide

VictoriaLogs Query Syntax (LogsQL)

Basic Queries

# Query by namespace
k8s_cluster:k8s-prod namespace:default

# Query by Pod name (supports regex)
k8s_cluster:k8s-prod pod:~"nginx.*"

# Keyword filter
k8s_cluster:k8s-prod namespace:production error

# Exact phrase match
k8s_cluster:k8s-prod "connection refused"

# Exclude pattern
k8s_cluster:k8s-prod error NOT "health check"

Advanced Queries

# Multi-condition combination
k8s_cluster:k8s-prod namespace:production container:app error OR exception

# Regex match log content
k8s_cluster:k8s-prod ~"OOM|OutOfMemory|killed process"

# Filter by container
k8s_cluster:k8s-prod pod:my-app container:sidecar

Log Analysis Workflow

Start with kubectl logs -n <namespace> <pod> --all-containers --since=1h --timestamps. Query VictoriaLogs through its documented HTTP endpoint with curl --get --data-urlencode; never interpolate untrusted query text into a shell command.

1. Error Log Analysis

1. Query error/exception logs from the last 1 hour
2. Group and count by error type
3. Identify error patterns (repeated, concentrated in a Pod)
4. Correlate Pod status and events
5. Provide root cause analysis and recommendations

2. Performance Log Analysis

1. Search for slow/timeout/latency related logs
2. Extract response time data
3. Identify common traits of slow requests (same endpoint, same DB query, etc.)
4. Correlate with resource usage at that time
5. Locate performance bottlenecks

3. Application Startup Log Analysis

1. Query Pod logs around recent restarts
2. Search for panic/fatal/killed/OOM keywords
3. Check startup order and dependency service connectivity
4. Analyze readiness/liveness probe failure logs

Cluster Label Mapping

Cluster Name	k8s_cluster Label
Production	k8s-prod
Compute	k8s-hpc
Tencent Cloud	tke-apcpa-prod
LLM	k8s-llm
Intelligent Voice	k8s-cce
Alibaba Cloud	k8s-ack

Log Collector Operations

fluent-bit Common Issues

• Memory overflow: Check buffer config, limit Mem_Buf_Limit
• Log loss: Check backpressure, increase Retry_Limit
• Parse failure: Check Parser config matches log format

vector Common Issues

• High CPU: Check for complex regex in transforms
• Connection drops: Check VictoriaLogs endpoint reachability
• Data delay: Check batch and buffer configuration

Keep collection config changes in Git, show the diff, and require approval before apply or rollout restart. Redact tokens, authorization headers, connection strings, and application secrets from logs.