aidp-rest-generic

# `aidp-rest-generic` — Generic REST via AIDP `aidataplatform` (`type=GENERIC_REST`) Read from arbitrary REST APIs as a Spark DataFrame. The connector requires a server-published **manifest** (a small JSON describing each API endpoint, parameters, and response schema) so it knows how to parse responses without a custom integration. ## When to use - Any REST endpoint that exposes a manifest URL (custom enterprise APIs commonly do). - Mentioned: "Generic REST", "manifest URL", "REST connector". ## When NOT to use - For **Fusion ERP/HCM/SCM** REST → [`aidp-fusion-rest`](../aidp-fusion-rest/SKILL.md). Different shape (no manifest; ≤499/page paging). - For **Fusion BICC** bulk extracts → [`aidp-fusion-bicc`](../aidp-fusion-bicc/SKILL.md). - For **EPM Cloud Planning** → [`aidp-epm-cloud`](../aidp-epm-cloud/SKILL.md). - For **Essbase** → [`aidp-essbase`](../aidp-essbase/SKILL.md). ## Read ```python import os from oracle_ai_data_platform_connectors.aidataplatform import ( AIDP_FORMAT, aidataplatform_options, ) opts = aidataplatform_options( type="GENERIC_REST", user=os.environ["REST_USER"], password=os.environ["REST_PASSWORD"], schema=os.environ.get("REST_SCHEMA", "default"), extra={ "base.url": os.environ["REST_BASE_URL"], # e.g. http://api.internal/v1 "manifest.url": os.environ["REST_MANIFEST_URL"], # e.g. http://api.internal/v1/manifest "auth.type": "basic", "api": os.environ["REST_API"], # e.g. "getOrdersByOrderID" # Any number of derived.property.<name> values feed into the API call: "derived.property.orderNo": os.environ.get("REST_ORDER_NO", "12345"), }, ) df = spark.read.format(AIDP_FORMAT).options(**opts).load() df.show(5) ``` ## Manifest contract The manifest describes: - `apis` — the named API operations (e.g. `getOrdersByOrderID`) - `parameters` — what the connector should send (path/query/body) - `responseSchema` — the Spark schema the connector should infer If you don't have a manifest URL, this connector won't work — fall back to the requests-based pattern in [`aidp-fusion-rest`](../aidp-fusion-rest/SKILL.md) and adapt for your API. ## Multiple derived properties Pass each as a separate `extra={}` key: ```python extra={ "base.url": "...", "manifest.url": "...", "auth.type": "basic", "api": "searchOrders", "derived.property.fromDate": "2025-01-01", "derived.property.toDate": "2025-12-31", "derived.property.status": "OPEN", } ``` ## Manifest from a workspace / volume path (`manifest.path`) If the manifest is a static file you've uploaded to your AIDP workspace or a Volume — instead of being served over HTTP — use `manifest.path` instead of `manifest.url`. Same shape, different source. Useful when the manifest is hand-authored or version-pinned alongside your notebook. ```python opts = aidataplatform_options( type="GENERIC_REST", user=os.environ["REST_USER"], password=os.environ["REST_PASSWORD"], schema="default", extra={ "base.url": os.environ["REST_BASE_URL"], "manifest.path": "/Volumes/myvol/manifests/orders_api.json", "auth.type": "basic", "api": "searchOrders", "derived.property.status": "OPEN", }, ) df = spark.read.format(AIDP_FORMAT).options(**opts).load() ``` The path can be: - `/Volumes/<catalog>/<schema>/<volume>/path/to/manifest.json` (AIDP Volume) - `/Workspace/Shared/.../manifest.json` (workspace file — works but FUSE-flaky) Volume paths are the preferred location. ## Gotchas - **`auth.type=basic` only.** If the API uses OAuth / API key headers / mTLS, this connector won't help — use the Python `requests` path. - **Manifest must be reachable from the AIDP cluster's VCN.** Egress restrictions apply. - **Schema `schema` option is the AIDP/Spark logical schema** for the resulting DataFrame, not a server-side one. Use `default` if unsure. - **Paging** is handled by the connector based on the manifest. If the manifest declares `maxPageSize`, the connector batches automatically. ## References - Helper: [scripts/oracle_ai_data_platform_connectors/aidataplatform.py](../../scripts/oracle_ai_data_platform_connectors/aidataplatform.py) - Official sample: [oracle-samples/oracle-aidp-samples → `data-engineering/ingestion/Read_Only_Ingestion_Connectors.ipynb`](https://github.com/oracle-samples/oracle-aidp-samples/blob/main/data-engineering/ingestion/Read_Only_Ingestion_Connectors.ipynb)