| name | team-info |
| description | ObsInt Processing team reference — repos, services, deployment, clusters, shared coding standards, PR rules, and language conventions. Shared guidance for team repositories; repo-specific details live in each repo's AGENTS.md. |
ObsInt Processing — Team Info
Data Flow
Internal Pipeline
Insights Operator archives arrive on the platform.upload.announce Kafka topic. archive-sync (an insights-ccx-messaging consumer) downloads the archive to S3 (rh-openshift-obs-stage / rh-openshift-obs-prod-secure) and publishes to ccx.archive.synced. rules-processing (a data-pipeline consumer) picks up from ccx.archive.synced, downloads the archive from S3, runs OCP rules, and publishes results to ccx.insights.rules.results. rules-uploader writes each result back to S3. parquet-factory also reads from ccx.insights.rules.results and aggregates rule-hit and cluster-info data into Parquet files on S3 (hourly).
External Pipeline
ccx-data-pipeline (data-pipeline repo) consumes new archive events from Kafka, reads the archive from S3, runs OCP rules, and publishes results to ccx.ocp.results. insights-results-db-writer (same codebase as aggregator, different deployment params) consumes from ccx.ocp.results and writes to PostgreSQL. insights-results-aggregator reads from that DB and serves results via REST API. insights-results-smart-proxy is the API gateway — external clients (UI, integrations, notifications) go through it to reach the aggregator and content-service (rule metadata: titles, remediations, tags). ocp-advisor-frontend is the React UI on console.redhat.com.
ccx-notification-writer also consumes from ccx.ocp.results and writes new/changed hits into a notification DB. ccx-notification-service reads that DB and sends email/webhook notifications.
ccx-upgrades-data-eng queries RHOBS for cluster metrics and calls ccx-upgrades-inference (ML model) to predict upgrade failure risk.
Kafka
AWS MSK cluster managed by AppSRE, shared across tenants. Topics are configured via PRs to platform-mq. Used topics: platform.upload.announce, ccx.archive.synced, ccx.insights.rules.results, ccx.ocp.results.
Tech Stack
| Area | Technologies |
|---|
| Backend services | Go (aggregator, smart-proxy, notification-*, content-service, cleaner, exporter, gathering-conditions-service), Python (data-pipeline, parquet-factory, insights-ccx-messaging, upgrades-data-eng, upgrades-inference, template-renderer) |
| Frontend | React, PatternFly (ocp-advisor-frontend) |
| Messaging | Kafka (all inter-service communication) |
| Storage | PostgreSQL (aggregator DB, notification DB), S3 (Parquet files) |
| Deployment | Clowder/ClowdApp on OpenShift, managed via app-interface SaaS files |
| CI/CD | Konflux (Tekton), GitHub Actions, Jenkins |
| Testing | BDD with Behave (insights-behavioral-spec), IQE (iqe-ccx-plugin), pytest (Python services), Go standard testing |
All new code requires tests and coverage. See each repo's AGENTS.md (when present) or README for repo-specific setup and how to run locally / in ephemeral.
Deployment
app-interface
app-interface controls service definitions, deployments, namespaces, cluster config, Grafana dashboards, and SaaS promotions. Our services live under data/services/insights/ccx-data-pipeline/.
Browse: https://gitlab.cee.redhat.com/service/app-interface/-/tree/master/data/services/insights/ccx-data-pipeline
App-interface GraphQL API: https://app-interface.apps.rosa.appsrep09ue1.03r5.p3.openshiftapps.com/graphql
Visual app-interface: https://visual-app-interface.devshift.net
Available app-interface skills.
Repositories
External Data Pipeline (EDP)
| Repo | What it does | Related Repos | AGENTS.md |
|---|
| data-pipeline | Consumes archives from Kafka, runs OCP rules, publishes results. | insights-ccx-messaging, parquet-factory | AGENTS.md |
| insights-results-aggregator | Stores OCP rule hits in PostgreSQL, serves them via REST API to smart-proxy. Also deployed as insights-results-db-writer (same code, different params — writer consumes Kafka, aggregator serves RESTAPI). | insights-results-smart-proxy, aggregator-cleaner, aggregator-exporter | AGENTS.md |
| insights-results-smart-proxy | API gateway between external clients (UI, integrations) and backend pipeline services. | insights-results-aggregator, content-service, ocp-advisor-frontend | AGENTS.md |
| ccx-notification-service | Reads new / changed rule hits and sends notifications to customers. | ccx-notification-writer, insights-results-aggregator | — |
| ccx-notification-writer | Consumes rule-hit from Kafka and writes them into the notification DB. | ccx-notification-service | — |
| ccx-upgrades-data-eng | Queries RHOBS for cluster metrics and calls the inference model to predict upgrade risk. | ccx-upgrades-inference, obsint-mocks | — |
| ccx-upgrades-inference | REST API serving the ML model that predicts upgrade failure likelihood from cluster metrics. | ccx-upgrades-data-eng | — |
| content-service | Serves rule metadata (titles, descriptions, remediations, tags, groups) to smart-proxy. | insights-results-smart-proxy, ocp-advisor-frontend | — |
| insights-content-template-renderer | Renders report messages from DoT.js templates using content data and report details. | content-service | — |
| insights-results-aggregator-cleaner | Periodic job that deletes old/obsolete records from the aggregator database. | insights-results-aggregator | — |
| insights-results-aggregator-exporter | Exports aggregator PostgreSQL data to S3 as CSV for offline analysis. | insights-results-aggregator | — |
| insights-operator-gathering-conditions | Remote configuration data (conditional gathering rules, rapid recommendations) for Insights Operator. | insights-operator-gathering-conditions-service | — |
| insights-operator-gathering-conditions-service | Serves gathering conditions and rapid recommendations to Insights Operator on clusters. | insights-operator-gathering-conditions | — |
Frontend
| ocp-advisor-frontend | React UI for OCP Advisor on console.redhat.com — displays rule hits per cluster. | insights-results-smart-proxy, content-service | — |
Internal Data Pipeline (IDP)
| Repo | What it does | Related Repos | AGENTS.md |
|---|
| insights-ccx-messaging | Python framework (on top of insights-core-messaging) that all CCX Kafka consumers inherit from. | data-pipeline, parquet-factory | AGENTS.md |
| parquet-factory | Aggregates rule-hit events from Kafka into Parquet files and uploads them to S3. | data-pipeline, insights-ccx-messaging | — |
Insights on Prem
| Repo | What it does | Related Repos | AGENTS.md |
|---|
| insights-on-prem | Deployment files and code for running Insights on Prem. | all EDP ones | AGENTS.md |
Shared Tooling and Testing
| Repo | What it does | Related Repos | AGENTS.md |
|---|
| processing-tools | Scripts, CI workflows, skills, and automations shared across all team repos. | all repos | — |
| ccx-docs | Internal documentation for the team. | all repos | AGENTS.md |
| insights-operator-utils | Shared Go library used by multiple insights-operator-* and pipeline services. | insights-results-aggregator, smart-proxy, gathering-conditions-service | — |
| insights-behavioral-spec | BDD test suite (Gherkin + Python) covering the full pipeline, OCM, notifications, and ServiceLog. | all pipeline services | AGENTS.md |
| iqe-ccx-plugin | IQE test plugin for CCX integration tests in ephemeral environments. | insights-behavioral-spec | AGENTS.md |
| obsint-mocks | Mock AMS and RHOBS endpoints used for local dev and BDD testing. | ccx-upgrades-data-eng | — |
| insights-results-aggregator-mock | Mock aggregator/smart-proxy API for local frontend and BDD development. | insights-results-aggregator, ocp-advisor-frontend | — |
| insights-results-mcp | MCP server exposing Insights results to LLMs via the Model Context Protocol. | - | — |
| insights-results-aggregator-utils | Utility scripts (Python/Go) for testing and interacting with pipeline REST APIs. | insights-results-aggregator, insights-results-smart-proxy | — |
Deployment Flow
All services are deployed via app-interface SaaS files:
- Stage — SaaS target uses
ref: master (or main). Merging to the default branch auto-deploys to stage.
- Prod — SaaS target uses
ref: <pinned-sha> with promotion: auto: false. Promoting requires an app-interface MR that updates the sha (use /promote-hypershift in app-interface).
- Ephemeral — short-lived environments for testing, deployed via Bonfire. See Playing with ephemeral for setup or example config.
Each SaaS file publishes/subscribes to promotion channels (e.g. commit-lag-exporter-*).
Where to update things in app-interface
Under data/services/insights/ccx-data-pipeline/:
| What | Path |
|---|
| External SaaS files | external-data-pipeline/<service>.yml |
| Internal SaaS files | internal-data-pipeline/<service>.yml |
| Stage namespace | namespaces/stage-ccx-data-pipeline-stage.yml |
| Prod namespace | namespaces/ccx-data-pipeline-prod.yml |
| Pipelines namespace | namespaces/ccx-data-pipeline-pipelines.appsrep09ue1.yaml |
| Grafana dashboards | provider: directory resourceTemplates in each SaaS file, path /dashboards |
| SLO document | slo-documents/ccx-data-pipeline.yml |
| Post-deploy jobs | jobs/post-deploy-jobs.yml |
Clusters & Environments
| Environment | Cluster | Namespace |
|---|
| Stage | crcs02ue1 | ccx-data-pipeline-stage |
| Production | crcp01ue1 | ccx-data-pipeline-prod |
| Pipelines (Konflux) | appsrep09ue1 | ccx-data-pipeline-pipelines |
IC Schedule
It may be relevant to know who is currently on watch duty. The rotation: data/teams/insights/schedules/ccx-processing-ic.yml in app-interface — browse
Related Skills (processing-tools)
The processing-tools repo hosts shared skills that can be installed into any agent:
| Skill | What it does |
|---|
konflux-dep-bumps | Triage and fix failing Konflux/MintMaker dependency bump PRs. |
resolve-cve | Resolve CVE vulnerability issues from Jira — assess, bump, or mark not-affected. |
update-refs | Update refs of deployments to the latest commit SHA |
To install a skill from processing-tools (Claude-specific):
npx skills add RedHatInsights/processing-tools --skill <skill-name> -g -a claude-code -y
claude plugin marketplace add https://github.com/RedHatInsights/processing-tools
claude plugin install <skill-name>
curl -sL https://raw.githubusercontent.com/RedHatInsights/processing-tools/master/skills/<skill-name>/SKILL.md \
-o ~/.claude/skills/<skill-name>/SKILL.md --create-dirs
Shared Standards
Conventions for all team repositories. Individual repos' AGENTS.md files should cover only what's specific to that repo.
Agent Conventions
When working on tasks in any team repository, create a TODO list to track progress and ensure all steps are completed systematically.
Pull Request Requirements
Branch protection and merge controls (CODEOWNERS, status checks, bot bypass) are documented in ProdSec and enforced via github-rulesets. Team policy is stricter than the org floor:
- Minimum 2 approvals from maintainers before merging (reviewers are defined in each repo's
CODEOWNERS file)
- Commit messages: short summary line, optionally followed by a blank line and a body with additional context or reasoning
- If related to a Jira task, include the ticket ID:
[CCXDEV-12345] Summary
- Focus on the "why" rather than the "what"
- PR description: what changed, why, how it was tested, and a Jira ID when applicable
- Unfinished PRs: use GitHub's draft PR feature to prevent accidental merging; if drafts are unavailable, prefix title with
[WIP]
- Breaking changes: must be documented and communicated to the team
- License headers must be present on source files
- No direct commits to the default branch — always use a feature branch
- CodeRabbit posts an initial summary/review on GitHub PRs — advisory only; it does not count toward the two required approvals, but authors should address useful findings
What reviewers should check
Human or agentic reviewers should understand the PR from its description, then verify:
- Changes match the description and linked Jira context
- Documentation is updated where needed (docstrings, README, or other)
- Tests and coverage expectations in Testing are met for the size of the change — larger functional impact needs more thorough testing and review (not just more lines/files)
- Full diffs are loaded for all files, including ones collapsed or hidden by default (supply-chain / sneaky-change risk)
Pre-Push Checklist
Go repos have a make before_commit (or before-commit) target that runs the full suite of checks — run it before pushing. Python repos use pre-commit hooks instead (pre-commit run --all-files). Check the repo's Makefile for the exact target.
Security
- Never log sensitive data: organization IDs must be sanitized, no auth tokens, no PII
- Credentials: use environment variables or config files, never hardcode
- Input validation: sanitize and validate all external input
- SQL injection: use parameterized queries, never string concatenation
Shell Scripts
All shell scripts must pass shellcheck. Go repos expose make shellcheck.
Build & Tooling
- Makefile is the standard task runner across all repos — build, test, lint, and pre-push checks are all make targets
- Behavioral tests: most services have BDD test coverage in insights-behavioral-spec
- Monitoring: Prometheus metrics and health endpoints (
/health, /ready) are standard for all deployed services
- Pre-commit hooks: this repo provides a shared
.pre-commit-config.yaml covering both Go and Python (golangci-lint, abcgo, ruff, shellcheck, hadolint). It also exports custom hooks (abcgo, go-version-consistency) that repos can use via repo: https://github.com/RedHatInsights/processing-tools
- Shared CI workflows: linter, test, and BDD workflow templates are synced to repos via
.github/sync.yml
Go Standards
- Go version: 1.22+
- Formatting:
gofmt
- Linting:
golangci-lint (specific linter set configured per repo)
- Documentation: GoDoc comments required for all exported symbols
- Error handling: return errors explicitly, no panics in production; use custom error types
- Logging:
rs/zerolog with structured context and request IDs
- Testing: standard
go test; check each repo's Makefile for integration test targets
Python Standards
- Line length: 100 characters
- Linter + formatter:
ruff
- Docstrings: Google style, required for all public methods
- Error handling: custom exception classes, always log before raising, structured logging via
logging.getLogger(__name__)
- Testing:
pytest; check each repo's Makefile for test targets and configuration
Testing
Normative rules for PRs; strategy and test-type detail live in Quality.
- All changes need tests appropriate to the change (unit in-repo; BDD / IQE / higher-level suites as described in Quality)
- Coverage must not decrease — tracked with Codecov on repos that have unit tests. There is no fixed %-gate required on every repo; Codecov / related checks fail when coverage drops vs the base branch
- How to run tests: see the repo's
AGENTS.md or Makefile; frameworks are listed under Tech Stack above
- Bigger functional changes need correspondingly stronger test coverage (see also What reviewers should check)
