remote-logging

name	remote-logging
description	Ship mobile logs to a remote backend (Datadog, Grafana Loki, Elastic, OTel collector) with batching, sampling, PII redaction, offline buffering, and backpressure. Use when introducing or reviewing remote log shipping.

Remote Log Shipping on Mobile

Instructions

Mobile clients are unreliable log sources: networks flap, batteries die, users background the app. Ship logs defensively: batch, sample, redact, and never block the UI thread.

1. Pick a Transport

• OTel log exporter (OTLP/HTTP) to a collector you own. Most flexible; collector handles fan-out to Loki / Datadog / Elastic.
• Vendor SDK (Datadog Logs SDK, Sentry logs, Grafana Faro). Less config, more lock-in.
• Raw HTTPS POST to your own endpoint. Only reasonable if you already run an ingest service.

Prefer OTLP so you can re-route without touching the client. Send via HTTP/protobuf, not gRPC (better behavior on constrained mobile networks).

2. Batch and Schedule

• Batch size: up to 100 records or 64 KB, whichever first.
• Flush interval: every 30 s in foreground, 120 s in background (iOS background URLSession / Android WorkManager).
• Never flush on UI thread. All work on a dedicated dispatcher (Kotlin) / queue (Swift) / isolate (Dart) / native bridge async task (RN).
• Coalesce: dedupe identical log lines within a flush window; increment attrs.count instead of sending N copies.

3. Sampling

• Sample by logger name and level: 100% of error/fatal, 10-20% of info, 1-5% of debug, 0% of trace.
• Per-user-per-day budget: cap at 5000 log records / user / day; after that, keep errors only.
• Always keep 100% of logs that share a trace with a span marked error=true (tail sampling at the collector if possible).

fun shouldKeep(entry: LogEntry): Boolean = when (entry.level) {
    "error", "fatal" -> true
    "warn"           -> Random.nextDouble() < 0.5
    "info"           -> Random.nextDouble() < 0.15
    "debug"          -> Random.nextDouble() < 0.02
    else             -> false
}

Log the sample rate alongside the record as attrs.sample_rate so the backend can re-inflate counts.

4. Offline Buffering

• Persist unsent batches to a bounded on-disk ring buffer (e.g. SQLite table, Dart hive, iOS file-based queue).
• Cap the buffer (e.g. 10 MB / 50 000 records). Drop oldest on overflow and emit a meta-log remote_log_overflow.
• Encrypt the buffer at rest if logs can contain anything sensitive: EncryptedSharedPreferences / DataStore with MasterKey on Android, Keychain-wrapped key on iOS.
• Clear the buffer on sign-out if logs can be tied to the signed-out user.

5. Backpressure

• If the backend returns 429 or 5xx, apply exponential backoff (base 2 s, max 5 min, full jitter).
• If backoff count exceeds a threshold, pause shipping and mark a Session.logsPaused = true so feature code can skip expensive logs.
• Do not retry forever on 400/401 -- drop the batch and alert in-app telemetry (see metric-alerts).

6. PII and Redaction in Transit

• Re-run the redactor at the shipper boundary even if already run at emit -- defense in depth.
• Strip HTTP Authorization headers, cookies, and body payloads from instrumented network logs.
• Redact stack trace filenames that contain user directories (/Users/alice/...). Replace with ~/.
• TLS is mandatory. Pin certificates for your own ingest endpoint; do not pin to vendor endpoints that rotate certs.

7. Example: Android OTel Log Exporter

val logExporter = OtlpHttpLogRecordExporter.builder()
    .setEndpoint("https://otel.example.com/v1/logs")
    .addHeader("Authorization", "Bearer ${BuildConfig.OTEL_TOKEN}")
    .setTimeout(Duration.ofSeconds(10))
    .build()

val logProvider = SdkLoggerProvider.builder()
    .addLogRecordProcessor(
        BatchLogRecordProcessor.builder(logExporter)
            .setMaxExportBatchSize(100)
            .setScheduleDelay(Duration.ofSeconds(30))
            .setMaxQueueSize(2048)
            .build()
    )
    .setResource(Resource.create(Attributes.of(
        AttributeKey.stringKey("service.name"), "my-app",
        AttributeKey.stringKey("service.version"), BuildConfig.VERSION_NAME,
        AttributeKey.stringKey("deployment.environment"), BuildConfig.FLAVOR
    )))
    .build()

8. iOS Background URLSession

let config = URLSessionConfiguration.background(withIdentifier: "logs.upload")
config.sessionSendsLaunchEvents = false
config.isDiscretionary = true
config.allowsCellularAccess = true
logsSession = URLSession(configuration: config, delegate: self, delegateQueue: nil)

iOS will schedule uploads when the device is charging and on Wi-Fi if isDiscretionary = true -- good for non-urgent logs, bad for error logs. Keep a parallel foreground session for errors.

9. Cost and Cardinality

• Cardinality killers: adding user_id_hash, device_model, or session_id as indexed attributes. Keep them as unindexed fields.
• Do not log once per frame, once per scroll event, or once per network byte.
• Review top loggers by ingested volume weekly; anything > 1% of total volume needs justification.

10. Verification

• Unit test: feeding 10k records generates <= 100 batches of <= 64 KB.
• Integration test: airplane mode for 5 minutes then reconnect drains the queue without loss up to the buffer cap.
• Chaos test: return 503 for 60 seconds; shipper backs off and eventually succeeds.

Checklist

• All shipping runs off the UI thread on a dedicated worker.
• Batches are bounded by size and time; flush interval differs foreground vs background.
• Sampling by level + per-user-per-day budget is implemented; sample_rate is attached.
• Offline buffer is bounded, encrypted if sensitive, and cleared on sign-out.
• Backoff on 429/5xx with jitter; 4xx drops the batch.
• Redaction runs both at emit and at the shipper boundary.
• TLS enforced; cert pinning on owned endpoints.
• Cost/cardinality review is part of the release checklist.