https-outcalls

# HTTPS Outcalls ## What This Is HTTPS outcalls allow canisters to make HTTP requests to external web services directly from on-chain code. In the default **replicated** mode, every node on the subnet makes the same request and all of them must agree on the response. A transform function strips non-deterministic fields (timestamps, request IDs, ordering) so that every replica sees an identical response and can reach consensus. Two other modes avoid that agreement step: **non-replicated** (`is_replicated = false`, one node makes the request) and **flexible** (`flexible_http_request`, a committee of nodes return their individual responses). See [Outcall modes](#outcall-modes). ## Support matrix | | Rust | Motoko | |---|---|---| | Package | `ic-cdk` **0.20.3+** with `ic-cdk-management-canister` **0.2+** | `ic` **4.x** (mops), which requires `core` **2.5.0+** and `moc` **1.4.0+** | | Entry point | `HttpRequest::new(url)` builder, `.send()` | `Call.httpRequest(args)` | | Pricing version 1 (legacy) | only by pinning `ic-cdk-management-canister` **0.1**, or calling `aaaaa-aa` directly with `ic_cdk::api::cost_http_request` | **yes**, this is what `Call.httpRequest` does | | Pricing version 2 (pay-as-you-go) | **yes**, the builder always selects it | — (**not available**, needs an unreleased `moc`) | | `flexible_http_request` | **yes**, `FlexibleHttpRequest::new(url)` | — (**not available**, same reason) | | `subnet_self_node_count` | `ic_cdk::api::subnet_self_node_count()` | — (**not available**, same reason) | Read this before writing code; what is available depends on the language and the dependency versions in the project. **Motoko code today is version 1 only**, and consistently so. The published `ic` 4.x `HttpRequestArgs` has no `pricing_version` field, Candid omits the absent optional, and the replica reads it as version 1 — which is exactly what `Call.httpRequest` funds. Nothing to work around; just do not expect version 2 economics from Motoko yet. For Rust the right code depends on which of three worlds the project is in. `ic-cdk` 0.19 has `ic_cdk::management_canister::http_request(&args)` (version 1; the module was removed in 0.20). `ic-cdk` 0.20 with `ic-cdk-management-canister` 0.1 has `ic_cdk_management_canister::http_request(&args)` (version 1). `ic-cdk` 0.20.3+ with `ic-cdk-management-canister` 0.2 has the builders, and is **version 2 only**: 0.2.0 removed the free `http_request` and the builder hard-codes `pricing_version: Some(2)` with no opt-out, so upgrading the crate *is* the migration. Rust also needs `serde_json` for JSON parsing. **Which version to write.** Version 1 is deprecated and version 2 is the direction, so *new* Rust code should use the 0.2 builders. But do **not** bump a project's pinned versions in order to migrate it as a side effect of an unrelated task: 0.2 is a breaking change (it deletes the free `http_request` and `HttpRequestArgs` gains a required field), so the upgrade is the caller's decision. Write correct code for the line the project is actually on, and tell them the upgrade exists. ## Canister IDs HTTPS outcalls use the IC management canister: | Name | Canister ID | Used For | |------|-------------|----------| | Management canister | `aaaaa-aa` | The `http_request` and `flexible_http_request` management call targets | You do not deploy anything extra. The management canister is built into every subnet. ## Mistakes That Break Your Build 1. **Forgetting the transform function.** In **replicated** mode, without a transform the raw HTTP response often differs between replicas (different headers, different ordering in JSON fields, timestamps). Consensus fails and the call is rejected. ALWAYS provide one there. In **non-replicated** and **flexible** mode it is optional *for consensus*, because each node's own response is delivered. Whether to set one there is a tradeoff. **No transform reserves nothing for it** (the instruction term defaults to 0, not to the query limit), which is the cheapest configuration available. Adding one defaults that term to the full query limit, so you must also declare `with_expected_transform_instructions`. What it buys back is the delivery fee, charged per byte on the **transformed** size: every byte stripped lowers the bill, so what decides it is how much there is worth stripping (bulky headers yes, a response that is already mostly payload no). The reservation floors that term at about 1 KB, so `get_cost()` understates the saving on a small response. Two non-cycle reasons can decide it anyway: in flexible mode the combined 2 MiB budget is measured after the transform, so smaller responses mean fewer get dropped, and reconciling is impossible if per-node noise makes every response distinct. 2. **Not attaching cycles to the call.** On a normal Application subnet, HTTPS outcalls are not free — the calling canister must attach cycles to cover the cost, and attaching zero fails the call. Both Motoko and Rust have wrappers that compute and attach the required cycles automatically: in Motoko, use `await Call.httpRequest(args)` from the `ic` mops package (`import Call "mo:ic/Call"`); in Rust, use the `HttpRequest` builder's `.send()` from `ic-cdk-management-canister` 0.2. Which pricing version you get depends on the wrapper: Motoko's `Call.httpRequest` prices version 1 with `ic0.cost_http_request(request_size, max_response_bytes)`, while Rust's `HttpRequest::send()` prices version 2 with `ic0.cost_http_request_v2`. Both are cost-schedule aware, so the same wrapper attaches the correct amount on any subnet type. **Under version 2 the up-front check is against the base fee only**, so an under-sized attachment is *not* rejected at call time: the call runs with tighter per-node limits and can fail partway, after the remote server was already contacted. See `references/pricing-version-2.md`. On a **cloud engine** (`CloudEngine` subnet), that amount is always 0 by design; do not "fix" a working outcall by attaching a hardcoded non-zero fee there — see the `cloud-engine-canisters` skill. 3. **Using HTTP instead of HTTPS.** The IC only supports HTTPS outcalls. Plain HTTP URLs are rejected. The target server must have a valid TLS certificate. 4. **Sizing `max_response_bytes` against the expected body — the limit is not body-only.** The spec defines the size of an HTTP request or response as *the total number of bytes representing the names and values of HTTP headers and the HTTP body*. Response **headers count against `max_response_bytes`**, and a real API commonly sends 1–2 KB of response headers (a unique request id, `Date`, the rate-limit family, CDN headers) before a single byte of body — against a tight cap that is a large share of the budget. Size the cap for **headers + body as they arrive from the server**, then add margin. The failure mode is total: the call fails every time rather than returning a truncated response. The maximum is 2MB = `2_000_000` bytes (decimal, not 2^21), and the same headers-plus-body definition caps the **request** you send at `2_000_000` bytes. 5. **Expecting the transform function to shrink an oversized response under the cap.** The cap is enforced **twice**: once on the raw response as it arrives from the server (headers first, then the body against what remains), and again on the transform's Candid-encoded **output**, which includes serialization overhead. Stripping headers in the transform cannot rescue a raw response that already exceeded the cap, because that first check fails before the transform ever runs. It *can* keep the transform's own output under the cap — worth doing when the transform would otherwise echo the headers back and its encoded output would exceed the limit. Both checks compare against the **same** `max_response_bytes` value, which is why a raw response that only just fits can still fail after the transform: the Candid overhead is added on top. So: size `max_response_bytes` for the raw response, and keep the transform's output small; never set a tight cap on the theory that stripping headers afterwards will make an oversized response fit. 6. **Ignoring the header limits.** Independent of `max_response_bytes`, the spec caps HTTP requests and responses at **64 headers**, **8 KiB** per header name or value, and **48 KiB** for all header names and values combined. The URL must not exceed **8192** bytes. On the request side these are enforced when the replica decodes your arguments, so an over-limit request never leaves the subnet and fails with `InvalidManagementPayload` — e.g. `Deserialize error: The number of elements exceeds maximum allowed 64` — rather than with any HTTP-looking error. If you send no `user-agent` header the IC adds `user-agent: ic/1.0`, and that added header does not count toward these limits. 7. **Omitting `max_response_bytes`.** Under **version 1**, if you do not set it the system assumes the maximum (2MB) and charges cycles accordingly — roughly 20.85 billion cycles on a 13-node subnet. Under **version 2** it no longer sets the price, but it is still the default for the `raw_response_bytes` expectation, so omitting it reserves for 2MB: 34.0 billion cycles rather than 5.3 billion for a 4,000-byte cap. Either way, always set it to a reasonable upper bound for your expected response (see pitfall 4 for what counts toward it). 8. **Non-idempotent POST requests without caution.** Because multiple replicas make the same request, a POST endpoint that is not idempotent (e.g., "create order") will be called N times (once per replica, typically 13 on a 13-node subnet). Use idempotency keys, or design endpoints to handle duplicate requests, or set `is_replicated = ?false` (Rust: `.non_replicated()`), which has a single node send the request and removes the rate-limit pressure entirely — at the cost of trusting that node not to observe or modify the response. 9. **Not handling outcall failures.** External servers can be down, slow, or return errors. Always handle the error case. There are **two distinct timeouts**, and neither traps — both come back as rejects (in Motoko the `await` raises a catchable `Error`; in Rust the wrapper returns `Err`): - The remote server does not respond within **30 seconds**: `SysFatal`, message `Timeout expired`. - The subnet does not produce a response within **60 seconds**: `SysTransient`, message `Canister http request timed out`. This one is normally the retryable one; the exception is a version 2 call that is under-funded *and* missing a node's report (see `references/pricing-version-2.md`). 10. **Calling localhost or private IPs.** HTTPS outcalls can only reach public internet endpoints. Localhost, 10.x.x.x, 192.168.x.x, and other private ranges are blocked. 11. **Forgetting the `Host` header.** Some API endpoints require the `Host` header to be explicitly set. The IC does not automatically set this from the URL. 12. **Leaving the version 2 expectations unset.** Anything you do not declare is *reserved* at its maximum: a 60-second round trip, and a transform running to the full query instruction limit. For a 4,000-byte cap that is around 5.3 billion cycles held, nearly all of it the transform reserve, against about 112 million once you declare `with_expected_transform_instructions` and `with_expected_roundtrip_time_ms`. Declare those two. Leave `raw_response_bytes` alone, because the server decides it; lower `max_response_bytes` instead if the byte terms dominate. Declare `with_expected_transformed_response_bytes` when your transform bounds its own output, which is the one case where the cap cannot be the lever. The *charge* is unaffected by any of this unless the smaller budget actually cuts the call short. `references/pricing-version-2.md` has the figures, the per-resource reasoning, and the failure modes. 13. **Expecting to reach version 1 through `ic-cdk-management-canister` 0.2.** You cannot. The builder is the only path the crate offers, it hard-codes `pricing_version: Some(2)`, and there is no `with_pricing_version`. `HttpRequest::from_args` looks like the escape hatch and is not: it is the documented way to migrate an existing call site, and it **silently overwrites `pricing_version` with 2**, so args that deliberately set 1 change version when you pass them through it. To stay on version 1, pin `ic-cdk-management-canister` 0.1, or call `aaaaa-aa` directly and price with `ic_cdk::api::cost_http_request`, which is still present in 0.20.3. 14. **Setting `pricing_version` by hand and funding it with the wrong cost function.** The field and the attachment have to agree. Set `pricing_version = 2` while attaching a `cost_http_request` (version 1) amount and the call is *accepted*, because the up-front check is only the base fee — it then runs on a smaller per-node allowance than version 2 intended and can fail partway. The reverse, a version 1 call funded with a `cost_http_request_v2` amount, is rejected outright with `http_request request sent with <X> cycles, but <Y> cycles are required.`, because version 1 wants the whole `max_response_bytes` up front. Let the wrapper set both: Rust's `HttpRequest::send()` pairs version 2 with `cost_http_request_v2`, and Motoko's `Call.httpRequest` pairs version 1 with `cost_http_request`. Note also that the replica *filters* an unrecognised version to version 1 with no error, so a bogus value fails as a funding mismatch rather than as a validation error. 15. **Hardcoding `total_requests` for a flexible outcall.** `total_requests` must not exceed the subnet size, which differs per subnet (13 or 34 on mainnet). Derive it: ```rust // WRONG, even when you want five nodes: 5 may exceed the subnet size. let replication = ReplicationCounts { total_requests: 5, min_responses: 3, max_responses: 5 }; // RIGHT: ask for five, but never more than the subnet has. let total_requests = subnet_self_node_count().min(5); let min_responses = total_requests / 2 + 1; ``` Also handle **any** count between `min_responses` and `max_responses` in the success arm — fewer than `max_responses` is a normal success, not a degraded one — and check each response's `status` separately, because no node had to agree with any other. See `references/flexible-outcalls.md`. ## Outcall modes Replicated and non-replicated are selected by `is_replicated`; flexible is a separate method. | | Replicated (default) | Non-replicated (`is_replicated = false`) | Flexible (`flexible_http_request`) | |---|---|---|---| | Who sends it | All N nodes | One node chosen by the system | A committee of `total_requests` nodes | | What you get | One agreed response | That node's response | Between `min_responses` and `max_responses` responses | | Transform | Required in practice | Optional, often still worth it | Optional, often still worth it | | Pricing | Version 1 or 2 | Version 1 or 2 | Always version 2 | | Extra methods | `GET`, `HEAD`, `POST` | plus `PUT`, `DELETE`, `PATCH` | plus those when `total_requests`, `min_responses` and `max_responses` are all equal | | Risk | N simultaneous requests trip API rate limits | That node could observe or modify the response | Reconciling the responses is your job | Use replicated when you need the integrity guarantee consensus gives, non-replicated for rate-limited APIs and non-idempotent POSTs, and flexible when the data changes faster than nodes could ever agree on it. ## Implementation ### Motoko Import both the wrapper and the types from the `ic` mops package: `import Call "mo:ic/Call"` and `import IC "mo:ic/Types"`. `Call.httpRequest` computes and attaches the required cycles. **This is pricing version 1.** Motoko has no version 2 path and no flexible outcalls yet (see the support matrix). Everything below is correct and supported; it is simply the legacy pricing model. ```motoko import Blob "mo:core/Blob"; import Nat "mo:core/Nat"; import Text "mo:core/Text"; import Call "mo:ic/Call"; import IC "mo:ic/Types"; persistent actor { // Transform function: strips headers so all replicas see the same response for consensus. // MUST be a `shared query` function. public query func transform({ context : Blob; response : IC.HttpRequestResult; }) : async IC.HttpRequestResult { { response with headers = []; // Strip headers -- they often contain non-deterministic values }; }; // GET request: fetch a JSON API public func getIcpPriceUsd() : async Text { let url = "https://api.coingecko.com/api/v3/simple/price?ids=internet-computer&vs_currencies=usd"; let request : IC.HttpRequestArgs = { url = url; // Always set — omitting defaults to 2MB and charges accordingly. // Budget for response headers + body: the cap covers both, and the // transform cannot bring an oversized response back under it. max_response_bytes = ?(10_000 : Nat64); headers = [ { name = "User-Agent"; value = "ic-canister" }, ]; body = null; method = #get; transform = ?{ function = transform; context = Blob.fromArray([]); }; is_replicated = null; }; // Call.httpRequest computes and attaches the required cycles automatically let response = await Call.httpRequest(request); switch (Text.decodeUtf8(response.body)) { case (?text) { text }; case (null) { "Response is not valid UTF-8" }; }; }; // POST transform: also discards the body, because httpbin.org echoes the // sender's IP in "origin", which differs across replicas. public query func transformPost({ context : Blob; response : IC.HttpRequestResult; }) : async IC.HttpRequestResult { { response with headers = []; body = Blob.fromArray([]); }; }; // POST request: send JSON data public func postData(jsonPayload : Text) : async Text { let url = "https://httpbin.org/post"; let request : IC.HttpRequestArgs = { url = url; max_response_bytes = ?(50_000 : Nat64); headers = [ { name = "Content-Type"; value = "application/json" }, { name = "User-Agent"; value = "ic-canister" }, // Idempotency key: prevents duplicate processing if multiple replicas hit the endpoint { name = "Idempotency-Key"; value = "unique-request-id-12345" }, ]; body = ?Text.encodeUtf8(jsonPayload); method = #post; transform = ?{ function = transformPost; context = Blob.fromArray([]); }; is_replicated = null; }; // Call.httpRequest computes and attaches the required cycles automatically let response = await Call.httpRequest(request); if (response.status == 200) { "POST successful (status 200)"; } else { "POST failed with status " # Nat.toText(response.status); }; }; }; ```