metrics-pipeline

name	metrics-pipeline
description	MetricsService, RedisMetricsCollector, ProcessPhotoMetrics, metrics table, Redis stats, leaderboards, XP processing, and photo processing state (processed_at/fp/tags/xp).

Metrics Pipeline

MetricsService is the single writer for all metrics — MySQL time-series and Redis aggregates. Nothing else touches metric counters. This is the golden rule.

Key Files

• app/Services/Metrics/MetricsService.php — Single writer for MySQL + Redis
• app/Services/Redis/RedisMetricsCollector.php — Redis operations (stats, HLL, rankings, tags)
• app/Services/Redis/RedisKeys.php — All Redis key builders (single source of truth for naming)
• app/Listeners/Metrics/ProcessPhotoMetrics.php — Queued listener on TagsVerifiedByAdmin
• app/Events/TagsVerifiedByAdmin.php — Trigger event for metrics processing
• app/Enums/LocationType.php — Global(0), Country(1), State(2), City(3) with scope prefixes

Invariants

1. Single writer rule. Only MetricsService writes to the metrics table and Redis metric keys. No other code may increment/decrement counters.
2. Processing state is four columns: processed_at, processed_fp, processed_tags, processed_xp. A photo with processed_at = null has never affected aggregates.
3. Fingerprint-based idempotency. MetricsService diffs old processed_tags JSON against new summary and writes only non-zero deltas. Safe to call repeatedly on any photo.
4. Summary must exist before metrics fire. GeneratePhotoSummaryService::run() MUST be called before TagsVerifiedByAdmin dispatches. MetricsService reads from photo.summary.
5. Redis is a derived cache. Rebuildable from the metrics table. RedisKeys::* is single source of truth for key naming.
6. processed_xp must be INT UNSIGNED, not TINYINT. Overflow bug documented in migration 2026_02_23_182605.
7. Tags count excludes categories to avoid double-counting: tags_count = objects + materials + brands + custom_tags.
8. ProcessPhotoMetrics logs a warning when photo not found. If a photo is soft-deleted before the queued listener runs, ProcessPhotoMetrics::handle() logs a warning and returns gracefully. It does NOT throw or retry.

Patterns

How MetricsService processes a photo

// MetricsService::processPhoto() — called by ProcessPhotoMetrics listener
DB::transaction(function () use ($photo) {
    $photo = Photo::whereKey($photo->id)->lockForUpdate()->first();
    $metrics = $this->extractMetricsFromPhoto($photo);  // reads photo.summary
    $fingerprint = $this->computeFingerprint($metrics['tags']);

    // Skip if nothing changed (fingerprint + XP both match)
    if ($photo->processed_fp === $fingerprint &&
        (int)$photo->processed_xp === (int)$metrics['xp']) {
        return;
    }

    // Route to create (first time) or update (re-tag)
    if ($photo->processed_at !== null) {
        $this->doUpdate($photo, $metrics, $fingerprint);
    } else {
        $this->doCreate($photo, $metrics, $fingerprint);
    }
});

MySQL upsert across timescales and locations

Each photo writes up to 40 rows: 5 timescales (all-time, daily, weekly, monthly, yearly) x 4 location scopes (global, country, state, city) x 2 (aggregate user_id=0 + per-user user_id>0).

DB::table('metrics')->upsert($rows,
    ['timescale', 'location_type', 'location_id', 'user_id', 'year', 'month', 'week', 'bucket_date'],
    [
        'uploads' => DB::raw('GREATEST(uploads + VALUES(uploads), 0)'),
        'tags'    => DB::raw('GREATEST(tags + VALUES(tags), 0)'),
        // ... same for brands, materials, custom_tags, litter, xp
    ]
);

Uploads delta: +1 for create, 0 for update, -1 for delete. GREATEST(..., 0) prevents negative counters.

Redis operations happen after MySQL commit

private function updateRedis(Photo $photo, array $payload, string $operation): void
{
    DB::afterCommit(function () use ($photo, $payload, $operation) {
        RedisMetricsCollector::processPhoto($photo, $payload, $operation);
    });
}

Redis key patterns (cluster-safe with hash tags)

RedisKeys::global()           // {g}
RedisKeys::country($id)       // {c:$id}
RedisKeys::state($id)         // {s:$id}
RedisKeys::city($id)          // {ci:$id}
RedisKeys::user($userId)      // {u:$userId}

RedisKeys::stats($scope)             // $scope:stats (HASH: uploads, tags, litter, xp, ...)
RedisKeys::hll($scope)               // $scope:hll (HyperLogLog for contributor count)
RedisKeys::objects($scope)            // $scope:obj (HASH: object_id => count)
RedisKeys::ranking($scope, $dim)      // $scope:rank:$dim (ZSET)
RedisKeys::xpRanking($scope)         // $scope:lb:xp (ZSET: user_id => xp, for leaderboards)
RedisKeys::userBitmap($userId)        // {u:$userId}:bitmap (activity bitmap)

Where TagsVerifiedByAdmin fires

1. Non-school users tag a photo: AddTagsToPhotoAction::updateVerification() — dispatches immediately after summary + XP (both web and mobile use this path).
2. Teacher approves school photos: TeamPhotosController::approve() — dispatches per photo after atomic is_public = true update.

Delete flow (metrics reversal)

// MetricsService::deletePhoto() — called synchronously in controllers before soft-delete
// Reads processed_tags JSON, applies negative deltas, clears processed_* columns
$photo->update([
    'processed_at' => null,
    'processed_fp' => null,
    'processed_tags' => null,
    'processed_xp' => null,
]);

In Redis, the delete operation also prunes zero-XP members from leaderboard ZSETs:

$pipe->zIncrBy(RedisKeys::xpRanking($scope), -abs($metrics['xp']), (string)$userId);
$pipe->zRemRangeByScore(RedisKeys::xpRanking($scope), '-inf', '0');

This keeps Redis consistent with MySQL (which filters xp > 0).

Common Mistakes

• Writing metrics outside MetricsService. Never DB::table('metrics')->increment(...) or Redis::hincrby(...) directly.
• Dispatching TagsVerifiedByAdmin before summary generation. MetricsService reads photo.summary — null summary = zero metrics.
• Comparing processed_xp as TINYINT. Values above 127 overflow. Column must be UNSIGNED INT.
• Forgetting row locking. Always use Photo::whereKey($id)->lockForUpdate()->first() inside the transaction.
• Assuming Redis is source of truth. Redis is a cache. The metrics table is authoritative.
• Including categories in tags_count. Categories are groupings, not countable items. Only objects + materials + brands + custom_tags.
• Not logging when ProcessPhotoMetrics can't find the photo. ProcessPhotoMetrics logs a warning (Log::warning(...)) when the photo is not found (soft-deleted or missing). It does not throw — the job completes successfully to avoid retries.
• Using is_public as the upload metrics gate. UploadPhotoController gates recordUploadMetrics() on $team->isSchool(), NOT on $photo->is_public. Private-by-choice photos (non-school) still get immediate upload XP. Only school photos defer metrics to teacher approval.