google-cloud-storage-okf

# Google Cloud Storage OKF Skill Use the `google-cloud-storage-okf` skill to efficiently navigate, inspect, and extract targeted byte ranges from unstructured Google Cloud Storage (`gs://`) objects via progressive disclosure, instead of reading or dumping entire files into the context window. ## Open Knowledge Format (OKF) in GCS The [Open Knowledge Format (OKF)](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing) establishes an open standard for attaching self-describing, AI-ready semantic knowledge directly to data assets without mutating raw content. While OKF at the catalog/dataset level organizes knowledge across tables into concept documents, **`google-cloud-storage-okf` adapts OKF to object storage**: - **Native Embedding**: Stores the OKF manifest directly in the blob's **GCS Object Context** (`custom_contexts`), eliminating disconnected sidecar files. - **Progressive Disclosure**: Provides a structured 3-tier access pattern (`peek` → `list_regions` → `expand`) so AI agents can navigate and extract information from multi-GB files with minimal context window overhead. ## GCS Object Context vs. Custom Metadata In Google Cloud Storage (GCS), there is a fundamental distinction between **Object Context** and **Custom Metadata**: - **GCS Object Context (`custom_contexts` / `contexts.custom`)**: A dedicated, structured context layer in GCS designed for AI agent knowledge, semantic indices, and progressive disclosure manifests. **OKF manifests are targeted primarily at GCS Object Context.** - **GCS Custom Metadata (`metadata` / `x-goog-meta-*`)**: Traditional key-value pairs used for generic user tagging and legacy object attributes. Custom metadata is separate from Object Context. OKF keys (such as `okf.sum`, `okf.r.000`, `okf.created_at`, and `okf.src_generation`) are written directly to **GCS Object Contexts** (`custom_contexts`). ### Client SDK Compatibility & Architecture The skill provides dual-layer compatibility across different versions of the `google-cloud-storage` Python SDK: - **Newer / Latest SDK**: If the client installation exposes higher-level typed classes (`ObjectContexts`, `ObjectCustomContextPayload`), the skill uses them to populate and read `blob.contexts`. - **Standard / Older SDK**: If higher-level classes are not yet exported in the local SDK version, the skill drops down to direct REST JSON payload injection (`blob._properties["contexts"]` / `blob._patch_property`) and signals `blob._changes`. When `blob.patch()` executes, the SDK transmits the standard HTTP REST `PATCH` request to GCS. - **Read Compatibility**: The skill inspects both `blob.contexts` and raw server response fields in `blob._properties["contexts"]`, guaranteeing that Object Contexts remain 100% readable and writable across all customer environments without startup import failures. -------------------------------------------------------------------------------- ## Core Architecture & Progressive Disclosure GCS-OKF stores lightweight, self-contained Table of Contents (TOC) metadata directly inside **GCS Object Context** on the original blob itself without modifying or duplicating the underlying object data: - **Tier 0 (`peek`)**: High-level summary card to quickly gauge relevance. - **Tier 1/2 (`list_regions`)**: Semantic Table of Contents detailing contiguous regions with inclusive byte ranges, titles, and keywords. - **Tier 2/3 (`expand`)**: Precise byte-range reader retrieving targeted slices based on region boundaries. - **Generative Layer (`write_okf`)**: Persists newly generated OKF indices into GCS Object Context. ``` +-------------------------------------------------------------+ | 1. peek(gs_uri) | | -> Evaluate relevance & inspect existing Object Contexts | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 2. list_regions(gs_uri) | | -> Locate specific chapters/sections and byte boundaries | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 3. expand(gs_uri, start_byte, end_byte) | | -> Read exact byte-range slice into context | +-------------------------------------------------------------+ ``` -------------------------------------------------------------------------------- ## Setup (Read Before Any OKF Call) Every OKF verb is served by the `google-cloud-storage-okf` MCP server. If that server is not registered with your MCP client, the tools do not exist in your session and no OKF work is possible. ### Tool Availability Gate **Before the first OKF step of any task, you MUST confirm that the MCP tools `peek`, `list_regions`, `expand`, and `write_okf` are all present in your tool list.** They may carry a harness-specific server-name prefix; a prefixed name counts as present. - **All four present**: proceed with the OKF workflow. - **Any one missing**: do NOT start the workflow. Tell the user the server is not registered with this client, relay the **One-Time Setup** steps below, and **STOP**. Do not resume OKF work until all four tools are bound. ### Never Bypass the MCP Server **NEVER** import or execute `scripts/core.py` or `scripts/mcp_server.py` directly as a substitute for the MCP tools — for example `python3 -c "import core; core.peek(...)"`, or launching `mcp_server.py` as a module in a subshell and driving it by hand. The server MUST be registered with the harness so that every OKF call goes through MCP. ### One-Time Setup 1. Install the **whole skill directory**, including `scripts/`, wherever your harness discovers skills. A `SKILL.md`-only copy cannot run the server. 2. Install the server's two Python dependencies, the MCP SDK and the GCS client library, using the `python3` that will run the server: ```bash python3 -m pip install mcp google-cloud-storage ``` 3. Authenticate and pick a project. The server reads Application Default Credentials and needs a Google Cloud project to bill and quota its requests: ```bash gcloud auth application-default login gcloud config set project <project-id> ``` Instead of a machine-wide project, you can pass one to this server only through the `env` block in step 4. 4. Register the canonical server command with your MCP client, then restart the client. Use an absolute path for `<abs skill_dir>` and the same `python3` as in step 2; the `env` block is optional: ```json { "mcpServers": { "google-cloud-storage-okf": { "command": "python3", "args": ["<abs skill_dir>/scripts/mcp_server.py"], "env": {"GOOGLE_CLOUD_PROJECT": "<project-id>"} } } } ``` Each harness has its own place for this snippet, and your harness knows where that is. If the client reports that the server failed to connect, run the command from step 4 by hand and read the traceback; a missing dependency or an incomplete install shows up there directly. ### Credential, Project, and Permission Errors If any OKF tool returns an error about credentials, a missing project (for example `Project was not passed and could not be determined from the environment`), or permission denied (401 or 403), the environment is not set up, not the request. Relay the error verbatim together with the matching fix from step 3 and **STOP**: - Missing or expired credentials: `gcloud auth application-default login`. - Missing project: `gcloud config set project <project-id>`, or the `env` block in step 4. - Permission denied: the authenticated principal needs read access to the object, and `storage.objects.update` for `write_okf`; ask the user to grant it. Do **NOT** change the user's gcloud configuration or credentials yourself, do NOT guess a project, and do NOT retry the operation through `gcloud storage`, `gsutil`, or client libraries as a workaround. -------------------------------------------------------------------------------- ## MCP Tools Reference ### 1. Peek Tool (`peek`) Retrieves the Tier 0 summary of the GCS object from its Object Context. - **Args**: - `gs_uri`: Target GCS URI (e.g., `'gs://my-bucket/documents/report.pdf'`). - **Returns**: - High-level summary string if OKF Object Context exists. - Stale index warning if the underlying object generation changed. - Diagnostic `OKF_NOT_FOUND` error detailing remaining Object Context capacity and pre-existing Object Context entries. ### 2. List Regions Tool (`list_regions`) Retrieves the Tier 1/2 Table of Contents (region index) stored in the object's Object Context. - **Args**: - `gs_uri`: Target GCS URI. - **Returns**: - Formatted JSON string containing a list of region descriptors, each with: - `id`: Region identifier (e.g., `'r0'`, `'r1'`). - `start_byte`: 0-indexed starting byte offset (inclusive). - `end_byte`: 0-indexed ending byte offset (inclusive). - `title`: Descriptive section/chapter title. - `keywords`: List of relevant keywords/tags. - Stale index warning JSON (`{"warning": "...", "regions": []}`) if generation changed. - Diagnostic `OKF_NOT_FOUND` error if regions are missing. ### 3. Expand Tool (`expand`) Fetches a specific byte-range slice from the GCS object payload. - **Args**: - `gs_uri`: Target GCS URI. - `start_byte`: 0-indexed start offset (inclusive). - `end_byte`: 0-indexed end offset (inclusive). - **Returns**: - UTF-8 decoded text slice (or Base64-encoded string for binary data). - Stale index warning string if the underlying object generation changed since OKF index creation. ### 4. Write OKF Tool (`write_okf`) Writes an OKF manifest into GCS Object Context (`okf.r.000`, `okf.sum`, etc.). - **Args**: - `gs_uri`: Target GCS URI. - `summary`: High-level overview string of the object. - `regions_json`: JSON array string of region descriptors (e.g., `'[{"id": "r0", "start_byte": 0, "end_byte": 1024, "title": "Introduction", "keywords": ["overview"]}]'`). - **Grounded Byte Boundaries**: Offsets MUST be 0-indexed integers satisfying `0 <= start_byte <= end_byte < object_size`. Regions MUST be listed in ascending order by `start_byte` and cannot overlap (for inclusive byte ranges, next `start_byte` must be at least previous `end_byte + 1`). `write_okf` strictly validates against actual object size and rejects inverted, negative, out-of-order, overlapping, or out-of-bounds byte ranges. - `created_at`: (Optional) ISO 8601 timestamp string (defaults to current UTC time). - **Returns**: - Success message or error message if Object Context capacity is exceeded.