| name | logging-ops |
| description | Query and analyze Kubernetes and VictoriaLogs data with kubectl, curl, and LogsQL. Use for Pod logs, error patterns,
startup failures, timeouts, VictoriaLogs operations, fluent-bit or Vector collection issues, and trace correlation.
适用于日志查询、日志分析、VictoriaLogs 运维、错误排查和日志采集配置。
|
Log Operations Guide
VictoriaLogs Query Syntax (LogsQL)
Basic Queries
# Query by namespace
k8s_cluster:k8s-prod namespace:default
# Query by Pod name (supports regex)
k8s_cluster:k8s-prod pod:~"nginx.*"
# Keyword filter
k8s_cluster:k8s-prod namespace:production error
# Exact phrase match
k8s_cluster:k8s-prod "connection refused"
# Exclude pattern
k8s_cluster:k8s-prod error NOT "health check"
Advanced Queries
# Multi-condition combination
k8s_cluster:k8s-prod namespace:production container:app error OR exception
# Regex match log content
k8s_cluster:k8s-prod ~"OOM|OutOfMemory|killed process"
# Filter by container
k8s_cluster:k8s-prod pod:my-app container:sidecar
Log Analysis Workflow
Start with kubectl logs -n <namespace> <pod> --all-containers --since=1h --timestamps. Query VictoriaLogs through its documented HTTP endpoint with curl --get --data-urlencode; never interpolate untrusted query text into a shell command.
1. Error Log Analysis
- Query error/exception logs from the last 1 hour
- Group and count by error type
- Identify error patterns (repeated, concentrated in a Pod)
- Correlate Pod status and events
- Provide root cause analysis and recommendations
2. Performance Log Analysis
- Search for slow/timeout/latency related logs
- Extract response time data
- Identify common traits of slow requests (same endpoint, same DB query, etc.)
- Correlate with resource usage at that time
- Locate performance bottlenecks
3. Application Startup Log Analysis
- Query Pod logs around recent restarts
- Search for panic/fatal/killed/OOM keywords
- Check startup order and dependency service connectivity
- Analyze readiness/liveness probe failure logs
Cluster Label Mapping
| Cluster Name | k8s_cluster Label |
|---|
| Production | k8s-prod |
| Compute | k8s-hpc |
| Tencent Cloud | tke-apcpa-prod |
| LLM | k8s-llm |
| Intelligent Voice | k8s-cce |
| Alibaba Cloud | k8s-ack |
Log Collector Operations
fluent-bit Common Issues
- Memory overflow: Check buffer config, limit Mem_Buf_Limit
- Log loss: Check backpressure, increase Retry_Limit
- Parse failure: Check Parser config matches log format
vector Common Issues
- High CPU: Check for complex regex in transforms
- Connection drops: Check VictoriaLogs endpoint reachability
- Data delay: Check batch and buffer configuration
Keep collection config changes in Git, show the diff, and require approval before apply or rollout restart. Redact tokens, authorization headers, connection strings, and application secrets from logs.
