| name | aidp-rest-generic |
| description | Pull data from any REST API into a Spark DataFrame using the AIDP `aidataplatform` Generic REST connector. Use when the user has a non-Fusion / non-EPM / non-Essbase REST endpoint with a `manifest.url` describing the schema. Auth is HTTP Basic with derived properties driving query parameters. |
| allowed-tools | Read, Write, Edit, Bash |
aidp-rest-generic — Generic REST via AIDP aidataplatform (type=GENERIC_REST)
Read from arbitrary REST APIs as a Spark DataFrame. The connector requires a server-published manifest (a small JSON describing each API endpoint, parameters, and response schema) so it knows how to parse responses without a custom integration.
When to use
- Any REST endpoint that exposes a manifest URL (custom enterprise APIs commonly do).
- Mentioned: "Generic REST", "manifest URL", "REST connector".
When NOT to use
Read
import os
from oracle_ai_data_platform_connectors.aidataplatform import (
AIDP_FORMAT, aidataplatform_options,
)
opts = aidataplatform_options(
type="GENERIC_REST",
user=os.environ["REST_USER"],
password=os.environ["REST_PASSWORD"],
schema=os.environ.get("REST_SCHEMA", "default"),
extra={
"base.url": os.environ["REST_BASE_URL"],
"manifest.url": os.environ["REST_MANIFEST_URL"],
"auth.type": "basic",
"api": os.environ["REST_API"],
"derived.property.orderNo": os.environ.get("REST_ORDER_NO", "12345"),
},
)
df = spark.read.format(AIDP_FORMAT).options(**opts).load()
df.show(5)
Manifest contract
The manifest describes:
apis — the named API operations (e.g. getOrdersByOrderID)parameters — what the connector should send (path/query/body)responseSchema — the Spark schema the connector should infer
If you don't have a manifest URL, this connector won't work — fall back to the requests-based pattern in aidp-fusion-rest and adapt for your API.
Multiple derived properties
Pass each as a separate extra={} key:
extra={
"base.url": "...",
"manifest.url": "...",
"auth.type": "basic",
"api": "searchOrders",
"derived.property.fromDate": "2025-01-01",
"derived.property.toDate": "2025-12-31",
"derived.property.status": "OPEN",
}
Manifest from a workspace / volume path (manifest.path)
If the manifest is a static file you've uploaded to your AIDP workspace or a Volume — instead of being served over HTTP — use manifest.path instead of manifest.url. Same shape, different source. Useful when the manifest is hand-authored or version-pinned alongside your notebook.
opts = aidataplatform_options(
type="GENERIC_REST",
user=os.environ["REST_USER"],
password=os.environ["REST_PASSWORD"],
schema="default",
extra={
"base.url": os.environ["REST_BASE_URL"],
"manifest.path": "/Volumes/myvol/manifests/orders_api.json",
"auth.type": "basic",
"api": "searchOrders",
"derived.property.status": "OPEN",
},
)
df = spark.read.format(AIDP_FORMAT).options(**opts).load()
The path can be:
/Volumes/<catalog>/<schema>/<volume>/path/to/manifest.json (AIDP Volume)/Workspace/Shared/.../manifest.json (workspace file — works but FUSE-flaky)
Volume paths are the preferred location.
Gotchas
auth.type=basic only. If the API uses OAuth / API key headers / mTLS, this connector won't help — use the Python requests path.- Manifest must be reachable from the AIDP cluster's VCN. Egress restrictions apply.
- Schema
schema option is the AIDP/Spark logical schema for the resulting DataFrame, not a server-side one. Use default if unsure.
- Paging is handled by the connector based on the manifest. If the manifest declares
maxPageSize, the connector batches automatically.
References
