cloud-logging-cross-project-configuration

# Configuring Cross-Project Logging This skill describes how to use `gcloud` commands to configure Cloud Logging so that you store log data in a central location, regardless of the point of origin. The skill also describes how to query log data when that data is stored in multiple projects. > [!IMPORTANT] **Sandbox Network Limitation (CRITICAL for Agent Testing):** > During evaluation or in restricted sandboxed environments, network traffic to > GCP APIs is blocked. Do **NOT** run network discovery commands to find > resource names, project IDs, or organization IDs. Always use the exact project > IDs or placeholders provided in the user prompt or instructions, for example, > `{project_id}`, `{source_project_id}`, `{central_project_id}`. Assume these > resources exist and proceed directly with configuration commands. Running > these discovery commands will cause the execution to hang and timeout. ## Safety and Confirmation Tiers (CRITICAL) Before executing any commands on behalf of the user, you MUST adhere to the following safety tiers based on the action requested: 1. **Tier R: Read-Only** * **Description:** Commands that only read state or query logs. * **Example commands:** * `gcloud logging read` * `gcloud logging buckets list` * **Rule:** No confirmation needed. You may execute these commands immediately to gather information. 2. **Tier M: Mutation (Non-Billing)** * **Description:** Configuration modifications or free metadata creations that do not incur direct storage or billing costs and do not affect resource security/access policies. * **Example commands:** * `gcloud logging views create` * `gcloud logging views update` * `gcloud logging scopes create` * `gcloud logging buckets create` * **Rule:** No confirmation needed. You may execute these commands immediately to apply configurations. 3. **Tier B: Billing and Security-Sensitive Mutations (High-Risk)** * **Description:** Operations that create billing-inducing resources or integrations, or modify security and IAM access control policies (presenting a risk of privilege escalation). * **Example commands:** * `gcloud logging metrics create` * `gcloud logging links create` * `gcloud projects add-iam-policy-binding` * **Rule:** **Interactive confirmation required.** These commands create resources that incur billing costs or alter security access. You MUST present the exact, literal command and receive user confirmation before executing. NEVER execute in the same turn as asking. 4. **Tier D: Causes irreversible data loss** * **Description:** Actions that permanently discard or delete logs, for example sink exclusions. * **Example commands:** * `gcloud logging buckets delete` * `gcloud logging sinks update --add-exclusion` * **Rule:** **Explicit typed confirmation required.** These commands discard or delete logs immediately and irreversibly, or they may result in log data not being stored. You MUST ask for explicit typed confirmation, for example, "Yes, discard logs", and halt execution until the user replies. ## Decision Matrix: Centralized Storage vs. Distributed Storage with Read-Time Aggregation Use this decision matrix to evaluate and choose between **Centralized Storage** and **Read-Time Aggregation**. With centralized storage, log data is routed to one log bucket, regardless of where the data originates. You write queries against the centralized log bucket. With read-time aggregation, log data is stored by the resource where it originates. However, a single query aggregates the data by querying all resources. After you have determined the optimal architecture for handling cross-project logs, follow the corresponding configuration steps detailed below. | Criterion | Centralized Storage | Read-Time Aggregation | | :-------------------- | :----------------------- | :----------------------- | | **GCP Project Scale** | Scales to thousands of | Best for < 375 projects. | : : projects. : : | **Log Storage** | Consolidated in a single | Resides in originating | : : log bucket. : resources. : | **SQL Analytics** | Easy; unified querying | Hard; requires querying | : : via Observability : multiple log buckets. : : : Analytics. : : | **Access Control** | Scoped access via log | Requires IAM access to | : : views on the centralized : all views on resources : : : log bucket. : that store log data. : | **Configuration** | Options vary based on | Will not interfere with | : **Complexity** : Project, Folder, : bucket-based log-based : : : Organization structure. : metrics. : | **Cost** | Potential for duplicate | Cost-effective; no data | : : storage of log buckets : replication. : : : if exclusions aren't : : : : set. : : ## Architecture ### Centralized Storage (Log Routing) ```mermaid graph LR subgraph "Source Project(s)" Log[Resource Logs] --> Sink["Sink: route-to-central-project"] end subgraph "Central Project" Sink --> Bucket["Bucket: central-logs-bucket (us-central1)"] end ``` ### Read-Time Aggregation (Log Scopes) ```mermaid graph LR subgraph "Source Project 1" Log1[Resource Logs] --> Bucket1["Bucket: _Default"] end subgraph "Source Project 2" Log2[Resource Logs] --> Bucket2["Bucket: _Default"] end subgraph "Scoping Project (No Log Storage)" Scope["Log scope: central-query-scope"] Scope -.-> View1["_AllLogs View on Bucket1"] Scope -.-> View2["_AllLogs View on Bucket2"] end ``` ## Setup Steps: Centralized Storage (Log Routing) Use these steps to route logs from one or more source projects to a central log bucket in a project. Create or select the Google Cloud project that you will use for storing your log data. This is the central project. ### 1. Create log bucket in central project (Tier M) Create a custom log bucket with Log Analytics enabled. > [Tip] Use regional log buckets, for example, set the location to > `us-central1`. Don't use the `global` location. This approach ensures > compatibility with Observability Analytics and SQL querying. ```bash gcloud logging buckets create {bucket_id} \ --project={central_project_id} \ --location={region} \ --retention-days={retention_days} \ --enable-analytics ``` ### 2. Create log sink in central project (Tier M) Create a project-level sink in the central project pointing to the central log bucket. This sink routes logs that land in the central project's Log Router into the central log bucket. Assuming the newly created bucket is specifically for centralizing data from other projects (and the central project is not one of the source projects), the central sink should exclude the central project's own logs. Include the `_Required` audit log exclusions on this central sink to avoid redundancy between `{bucket_id}` and the `_Required` bucket. ```bash gcloud logging sinks create {sink_name} \ logging.googleapis.com/projects/{central_project_id}/locations/{region}/buckets/{bucket_id} \ --project={central_project_id} \ --exclusion=filter='logName:"projects/{central_project_id}/"' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")' ``` ### 3. Create log sink in source resource (Tier M) To route logs to the central project, you must create a log sink in each source organization, folder, or project. While you can configure a sink to route only a subset of logs using the `--log-filter` argument, recommended practice is to route all non-audit logs, and then restrict access or partition logs at the destination using custom Log Views on the centralized log bucket. * **For an organization-level log sink** ```bash gcloud logging sinks create {sink_name} \ logging.googleapis.com/projects/{central_project_id} \ --organization={source_organization_id} \ --include-children \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")' ``` * **For project-level log sinks** ```bash gcloud logging sinks create {sink_name} \ logging.googleapis.com/projects/{central_project_id} \ --project={source_project_id} \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \ --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \ --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")' ``` ### 4. Grant IAM permissions to sink writers (Tier B) > [!IMPORTANT] **Security Action (Tier B):** Granting IAM permissions changes > access control policy and must be explicitly confirmed by the user before > execution. To allow the source sinks to route logs to the central project's router, and to allow the central sink to write logs to the central bucket: 1. **Grant Logs Writer permission to the source sink:** Retrieve the `writerIdentity` of the source log sink and grant it `roles/logging.logWriter` on the central project. ```bash # Get the writer identity of the source sink gcloud logging sinks describe {sink_name} \ --project={source_project_id} \ --format="value(writerIdentity)" ``` The output is the `{source_writer_identity}` value (for example `serviceAccount:...`) for the following command: ```bash # Grant Logs Writer permissions on the central project gcloud projects add-iam-policy-binding {central_project_id} \ --member={source_writer_identity} \ --role=roles/logging.logWriter ``` 2. **Grant Bucket Writer permission to the central sink:** Retrieve the `writerIdentity` of the central log sink and grant it `roles/logging.bucketWriter` on the central project. ```bash # Get the writer identity of the central sink gcloud logging sinks describe {central_sink_name} \ --project={central_project_id} \ --format="value(writerIdentity)" ``` The output is the `{central_writer_identity}` value for the following command: ```bash # Grant Bucket Writer permissions on the central project gcloud projects add-iam-policy-binding {central_project_id} \ --member={central_writer_identity} \ --role=roles/logging.bucketWriter ``` ### 5. Create custom Log Views on the central bucket (Tier M) To partition logs or restrict access by log ID or project (since all logs were routed to the same central bucket), create custom Log Views on the central bucket. * **Filter by log ID:** ```bash gcloud logging views create {view_id} \ --bucket={bucket_id} \ --location={region} \ --project={central_project_id} \ --log-filter='LOG_ID("{log_id}")' ``` * **Filter by source project:** ```bash gcloud logging views create {view_id} \ --bucket={bucket_id} \ --location={region} \ --project={central_project_id} \ --log-filter='project_id="{source_project_id}"' ``` ### Verify Centralized Log Routing (Tier R) To verify that logs are being routed from the source projects to the central regional log bucket: 1. **Write a test log in the source project:** ```bash gcloud logging write {test_log_id} "Test log entry for verification" \ --severity=WARNING \ --project={source_project_id} ``` 2. **Read the test log from the central log bucket:** For regional log buckets, you **must** specify the `--view` flag. Because the default view `_Default` only contains logs matching the default filter, you should query the **`_AllLogs`** view or your custom log view on the central bucket: ```bash gcloud logging read 'logName:"projects/{source_project_id}/logs/{test_log_id}"' \ --bucket={bucket_id} \ --location={region} \ --view=_AllLogs \ --project={central_project_id} ``` -------------------------------------------------------------------------------- ## Setup Steps: Read-Time Aggregation (Log Scopes) Create or select a Google Cloud project that you will use for querying your log data. This is the scoping project. Use these steps to configure a log scope. Log scopes let you issue a query for log data that is stored in multiple projects. ### 1. Create a custom Log View (Optional but recommended) (Tier M) Create a Log View in the source project to restrict which logs are accessible. > [!IMPORTANT] **Gotcha:** Log View filters can only contain specific > restrictions. Refer to > https://docs.cloud.google.com/logging/docs/logs-views.md.txt#view-filter ```bash gcloud logging views create {view_id} \ --bucket={bucket_id} \ --location={region} \ --project={source_project_id} \