Skip to content
Sign in

storage

v0.2.5

Object storage across AWS S3, GCS, Cloudflare R2, and a native local filesystem backend. Streamed uploads, signed URLs, and observable object change events.

license: Apache-2.0Used in 1 kitblobgcsobject-storager2s3

Install

bash
iii trigger compose::add worker=storage

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

iii-config.yaml
- buckets:
    scratch:
      provider: local
  providers:
    local:
      data_dir: data/storage
      http:
        bind_address: 127.0.0.1:0

storage

Object storage for the iii engine over S3, GCS, R2, and a native local filesystem backend. Streamed uploads, signed downloads, and observable object-created / object-deleted events — all behind one bucket: name regardless of the cloud underneath.

S3 multipart controls

storage::multipartCapabilities takes only { "bucket": "<configured name>" } and returns { "provider": "s3", "contract": "sha256-composite-v2-list-optional" } for the S3/R2 backend, or contract: null for unsupported backends. Unknown or unavailable buckets fail normally. Before returning a supported contract, this metadata-only operation sends HeadBucket using the live configured backend: 404, denied credentials and a disconnected provider fail the call. It requires bucket-scoped s3:ListBucket permission for HEAD, not account-wide ListBuckets. It neither lists object contents nor creates uploads/objects, and returns no credentials. The contract identifies checksum-bound signing, optional listing checksums, mandatory checksums on every Complete part and SHA256/COMPOSITE HEAD support. It is independent of the package version and does not certify a provider's multipart implementation or write/checksum permissions. Consumers must still validate the actual provider's complete multipart contract before rollout.

The S3 backend supports multipart create, checksum-bound part signing, list, complete and abort, plus checksum-enabled object HEAD. These controls require a provider that implements the matching SHA-256 composite checksum contract; validate against the actual provider and version before enabling an application. Other backends return unsupported for these operations. Browser part bytes travel directly to storage; the caller owns authorization, session lifecycle and cleanup. Configure an incomplete-multipart lifecycle rule or an approved equivalent.

ListParts returns each part's number, size and opaque ETag; checksum_sha256 is optional and is omitted when the provider does not report it. A present checksum must be valid SHA-256. This does not relax UploadPart signing or Complete: completion parts still require their authorized SHA-256 checksums. Callers must compare listed identity, size and ETag (and checksum when present) against the closed manifest, then require the exact checksum-enabled SHA256 COMPOSITE HEAD before publishing, including recovery after a lost completion response.

Worker release versions are selected and published by Release Control from a verified source SHA and build manifest; see deployment operations. The version in Cargo.toml is source metadata, not proof that this code is published.

Install

iii call compose::add worker=storage

iii call compose::add resolves the worker and its dependencies, writes exact declarations to worker-compose.yaml, and reconciles the Compose project. The worker depends on configuration (where its settings live) and secrets (which resolves secret:// / env:// credentials, see Credentials).

Quickstart

Use inline RPCs only for genuinely small values. Normal files should go directly between the client and object storage through a signed upload or download endpoint.

import { registerWorker } from 'iii-sdk'

const iii = registerWorker(process.env.III_URL ?? 'ws://127.0.0.1:49134')

await iii.call({
  function_id: 'storage::putObject',
  payload: {
    bucket: 'uploads',
    key: 'u/1/profile.jpg',
    body_base64: tinyValueBase64,    // small values only; 10 MiB hard limit
    content_type: 'image/jpeg',
  },
})

const { url, expires_at } = await iii.call({
  function_id: 'storage::presignUrl',
  payload: {
    bucket: 'uploads',
    key: 'u/1/next.jpg',
    method: 'PUT',
    expires_in_seconds: 600,
    content_type: 'image/jpeg',      // pinned into the signature
  },
})

await fetch(url, {
  method: 'PUT',
  headers: { 'Content-Type': 'image/jpeg' },
  body: file,
})

const { body_base64, content_type } = await iii.call({
  function_id: 'storage::getObject',
  payload: {
    bucket: 'uploads',
    key: 'u/1/profile.jpg',
  },
})

await iii.call({
  function_id: 'storage::deleteObject',
  payload: {
    bucket: 'uploads',
    key: 'u/1/profile.jpg',
  },
})                                  // idempotent: returns { deleted: false } if absent

const { content_type, etag, size, last_modified } = await iii.call({
  function_id: 'storage::headObject',
  payload: {
    bucket: 'uploads',
    key: 'u/1/profile.jpg',
  },
})                                  // fetches metadata only — no body download

For a multipart upload to a native local bucket, use presignPost:

const { url, fields } = await iii.call({
  function_id: 'storage::presignPost',
  payload: {
    bucket: 'scratch',
    key: 'videos/demo.mp4',
    content_type: 'video/mp4',
    expires_in_seconds: 600,
  },
})

const form = new FormData()
for (const [name, value] of Object.entries(fields)) form.append(name, value)
form.append('file', file)
await fetch(url, { method: 'POST', body: form })

putObject and getObject serialize bytes as base64 inside an engine message. They buffer the complete value and base64 adds roughly 33% overhead. Both enforce a 10 MiB decoded hard limit per call, but they are intended for truly small values, not general file transfer. Use presignPost/a signed PUT for uploads and a signed GET from presignUrl for downloads. A client that cannot reach the provider (a phone outside the LAN of a local bucket, a GCS bucket without signing credentials) can read any object through getObject in ranges: { offset, length }, see Ranged reads.

From a Rust worker:

use iii_sdk::{register_worker, CallRequest, InitOptions};
use serde_json::json;

let iii = register_worker("ws://localhost:49134", InitOptions::default());

iii.call(CallRequest {
    function_id: "storage::putObject".into(),
    payload: json!({
        "bucket": "uploads",
        "key": "u/1/profile.jpg",
        "body_base64": file_b64,
        "content_type": "image/jpeg",
    }),
    action: None,
    timeout_ms: Some(5_000),
}).await?;

Configuration

The storage worker gets its live configuration from the configuration worker, not from a local file. On startup it:

  1. Registers its config schema with the configuration worker (configuration::register, id storage).
  2. Fetches the live, env-expanded config (configuration::get).
  3. Observes configuration::on-updated (its entry's configuration:updated events) and hot-reloads.

--config <path> is an optional seed: when given, the file is loaded and sent as initial_value the first time the schema is registered (no stored value yet). It is not the live source of truth — once a value exists in the configuration worker, that value wins.

Hot-reload scope

On a configuration:updated event the worker re-fetches the authoritative config from the configuration worker (it does not trust the event payload). Every setting is live: bucket additions/removals, providers, credentials, underlying names, notification sources, local data directory, and the local HTTP listener/public URL all apply without restarting the worker.

Buckets are applied one by one. A bucket whose backend cannot be built — a secret:// reference that does not resolve, a malformed field, a credentials JSON that does not parse — stays listed by storage::listBuckets with error / error_stage, and every call on it answers BUCKET_UNAVAILABLE {bucket, stage, message}; every other bucket keeps working. A failed bucket is retried every 15 s (a secrets worker that came up late), and right away when a secret it references changes. A reload rebuilds only the buckets whose entry or resolved credentials changed: the others keep their clients, and notification pollers whose source did not change keep running. A notification source that cannot be prepared (a Cloudflare token the queue refuses) leaves its buckets working without object events and is reported as notifications_error.

The native-local listener stays transactional: the worker binds a new bind_address before publishing anything, and a port that is taken rejects the whole update, leaving the previous configuration serving requests.

A fresh install with no configured buckets runs with zero backends until a bucket is configured.

Config shape

Each bucket pins a provider (s3 | gcs | r2 | local) and the credentials for that provider. Buckets without notifications: work fine for RPCs; they produce no object events, and an observer of one is refused.

providers:
  local:
    data_dir: data/storage             # relative to III_COMPOSE_DIR
    http:
      bind_address: 0.0.0.0:49200      # omit `http` to disable direct transfers
      public_url: http://10.0.0.42:49200 # browser-visible LAN/VPN/proxy URL

buckets:
  uploads:
    provider: s3
    bucket: my-app-uploads             # underlying cloud bucket
    region: us-east-1
    notifications:
      sqs_queue_url: https://sqs.us-east-1.amazonaws.com/123/my-app-uploads-events

  documents:
    provider: gcs
    bucket: my-app-documents
    # credentials_file: /etc/iii/gcs-sa.json   # required for presignUrl

  avatars:
    provider: r2
    bucket: avatars
    account_id: env://R2_ACCOUNT_ID
    access_key_id: secret://STORAGE_AVATARS_ACCESS_KEY_ID
    secret_access_key: secret://STORAGE_AVATARS_SECRET_ACCESS_KEY
    display: { icon: image, color: "#F38020" }

  photos:
    provider: s3                       # S3-compatible: MinIO, B2, Wasabi, Spaces…
    region: us-east-1
    endpoint_url: https://minio.example.com
    force_path_style: true
    access_key_id: secret://STORAGE_PHOTOS_ACCESS_KEY_ID
    secret_access_key: secret://STORAGE_PHOTOS_SECRET_ACCESS_KEY

  reports:
    provider: gcs
    credentials_json: secret://STORAGE_REPORTS_CREDENTIALS_JSON   # the service-account JSON itself

  scratch:
    provider: local
    bucket: scratch
    display:
      icon: hard-drive                 # a Lucide icon name
      color: "#3B82F6"                 # #RRGGBB

The map key (uploads) is the worker-facing bucket name handlers reference: [a-z0-9][a-z0-9_-]{0,62}. The nested bucket: is the underlying cloud bucket (default: the map key). They can differ. display is read only by clients (the console, the app) to draw the bucket.

Credentials and the secrets worker

Every credential field — access_key_id, secret_access_key, session_token, r2's account_id, gcs's credentials_json and r2's notifications.api_token — takes, besides a literal or a ${VAR} template:

  • secret://NAME — a value in the secrets worker's encrypted vault;
  • env://NAME — an environment variable the secrets worker reads (the project's .env, else its own environment).

NAME matches ^[A-Za-z_][A-Za-z0-9_.-]{0,127}$. The worker resolves them with secrets::resolve as consumer storage: the secret must list storage in its consumers (secrets::set {name, value, consumers: ["storage"]}), else the bucket is unavailable with error_stage: "secrets" and a message saying so. Resolved values never appear in the configuration, a reply, logs or traces, and every error message is scrubbed of the credentials used.

Resolved values are cached. The worker observes the secrets worker's secrets::on-changed with its internal storage::on-secret-change: a rotated, deleted or re-shared secret is re-read at once and only the buckets that use it are rebuilt (deleted → those buckets report the error; re-created → they come back). Without the event a cached value is re-read every 5 minutes, and a failed one every 10–15 seconds.

secret_access_key, session_token, credentials_json and notifications.api_token are sensitive: storage::getBucket shows their literals as "__redacted__", and storage::saveBucket / storage::testBucket refuse a literal inside bucket (it goes as a reference or in credentials). Literals and ${VAR} templates that are already stored keep working.

Per-provider notes

  • S3 — defaults to the AWS credential chain (env, ~/.aws, IMDS, IRSA). Override with access_key_id / secret_access_key / session_token only if the default chain doesn't fit.
  • GCS — defaults to ADC (GOOGLE_APPLICATION_CREDENTIALS, GCE metadata, gcloud auth application-default login). credentials_json (the service-account JSON itself, usually secret://…) takes precedence over credentials_file (a path on the engine host). presignUrl requires a service-account key (or the metadata server's identity on Google Cloud); without one, or with an endpoint_url override, GCS presigns return PRESIGN_UNSUPPORTED and listBuckets reports direct_downloads: false. Pub/Sub notifications still authenticate through ADC.
  • R2 — required: account_id, access_key_id, secret_access_key. Endpoint URL is derived automatically as https://{account_id}.r2.cloudflarestorage.com.
  • local — implemented directly by the worker. Object bytes and JSON metadata are persisted under data_dir; no sidecar or external binary is required. http is optional: omit it when only the small inline RPCs are needed. When enabled, bind_address controls the listener and public_url controls the base URL returned to clients. Set public_url whenever clients reach the worker through a LAN address, VPN, container port mapping, or reverse proxy. With port 0 and no public_url, the worker chooses a free loopback port and returns that address. The native listener includes browser CORS support for these signed transfers; cloud buckets need equivalent CORS rules on the provider when uploads originate in the Console.

Existing rustfs data directories are not imported automatically: the native backend uses its own object/metadata layout under data_dir. Export or copy existing objects before upgrading, then upload them into the native bucket.

Custom endpoints

S3, R2, and GCS bucket configs accept an optional endpoint_url field for self-hosted S3-compatible stores (MinIO, Ceph, SeaweedFS), staging environments, or local testing against fake-gcs-server.

buckets:
  scratch-self-hosted:
    provider: s3
    region: us-east-1
    endpoint_url: https://s3.internal.example.com
    bucket: scratch

R2 with endpoint_url set emits a tracing::warn! at startup — the field is fully functional but production R2 should omit it and let the worker derive the endpoint automatically.

Wiring notifications

Provider Config field(s) Setup
S3 notifications.sqs_queue_url SQS queue + bucket event config for s3:ObjectCreated:* / s3:ObjectRemoved:* + sqs:ReceiveMessage,DeleteMessage IAM on the queue ARN.
GCS notifications.pubsub_subscription gsutil notification create -t TOPIC -e OBJECT_FINALIZE,OBJECT_DELETE gs://<bucket> + roles/pubsub.subscriber on the subscription.
R2 notifications.queue_id + notifications.api_token Cloudflare Queue + R2 event notifications on the bucket + API token scoped queue:consume.
local (none) The worker calls the object functions immediately after a native local write/delete commits.

Other config keys and their defaults live in src/config.rs; wire-stable error codes returned by every RPC live in src/error.rs.

Object events

Function Args Called when Event
storage::on-object-created { bucket, handler_timeout_ms? } An object is written to a bucket with a notifications source (PUT, multipart complete, copy-in; every write to a local bucket). { bucket, key, size, etag, content_type, created_at, raw_event_id }
storage::on-object-deleted { bucket, handler_timeout_ms? } An object is removed from such a bucket. { bucket, key, deleted_at, raw_event_id }

Both functions are private: only this worker calls them, once per event its notifications source reports, and any worker may observe them. An observer names one worker bucket in args.bucket; handler_timeout_ms (default 60000) bounds how long its target may take per event. An observer of a bucket without a running notifications source is rejected with CONFIG_ERROR. A configuration change that removes a bucket, or its notifications:, rejects the observers of that bucket with CONFIG_ERROR as well; observers of a bucket whose source is configured but fails to start stay, and receive events once it runs again.

Delivery is custom and at-least-once. For each event, the worker calls the target of every observer of its bucket whose filters match, concurrently, and the target receives the event itself (the observer's capture does not apply). A target acks by returning null or { ack: true }. The upstream message (SQS, Pub/Sub, Cloudflare Queues) is acked only when every target acked; { ack: false }, an error or a timeout leaves it for redelivery. A call of either function answers its caller { acked }. With no observer of the bucket, the worker builds no event and acks the upstream message.

iii.registerFunction('uploads::on-created', async (event) => {
  await thumbnails.queue(event.bucket, event.key)
  return { ack: true }
})

iii.observe(
  { function_id: 'storage::on-object-created', args: { bucket: 'uploads', handler_timeout_ms: 30000 } },
  'uploads::on-created',
)
use iii_sdk::Source;
use serde_json::json;

iii.observe(
    Source::function("storage::on-object-deleted").args(json!({ "bucket": "uploads" })),
    "uploads::on-deleted",
);

Filters narrow an observer further, by the event's fields: { "input.key": { "$prefix": "avatars/" } } delivers only keys under avatars/, and an event it filters out counts as acked.

The worker itself observes two functions with internal handlers (metadata internal: true, denied to agents): configuration::on-updated, filtered to its entry's configuration:updated events, with storage::on-config-change, and the secrets worker's secrets::on-changed with storage::on-secret-change (the observer stays pending while the secrets worker is not running). Both ignore their payload's values and re-read the authoritative state.

storage::saveBucket and storage::testBucket accept literal credentials in their input, so they are registered with observable: false: observing one fails with NOT_OBSERVABLE.

R2 notifications v1 caveat: the Cloudflare Queues consume-from-outside REST API is the youngest of the four upstreams. The worker probes the consume endpoint at startup and surfaces CF_QUEUE_AUTH_FAILED for 401/403, so token misconfiguration is visible immediately. If you hit redelivery or auth-scope edge cases in production, file an issue — v1.1 will finalize the consume path.

Console explorer

When the worker is connected, it injects a storage page into the Console. Wide panels use three simultaneous levels—buckets, folders/objects, and object details—while narrow panels show one level at a time. The explorer supports folder breadcrumbs, pagination, metadata inspection, direct upload, signed download, copy-key, and delete actions. Uploads and downloads use the direct transfer endpoints instead of moving file bodies through an inline RPC.

For native local uploads, enable providers.local.http. Cloud uploads also require the provider bucket to permit the Console origin through CORS.

The page's configure action opens a storage-specific editor while the Console continues to own validation, dirty tracking, save, reset, and the unsaved-change guard. The editor separates native-local runtime settings from bucket mappings and presents only the fields relevant to Local, S3, GCS, or R2. Saved changes are applied live; no storage setting requires a worker restart.

Bucket management

Bucket editors (the console, the iii app) create and change buckets through four functions instead of rewriting the whole configuration. All four are denied to agents by iii-permissions.yaml (they rewrite which providers and credentials the worker uses, or send credentials where a candidate points). Their requests refuse unknown keys and wrong-typed fields with CONFIG_ERROR naming the field (keys starting with _ are the engine's own metadata and are ignored).

Where credentials may go

A save makes the live bucket send its credentials to its endpoint; a test sends them at once. Neither may become a way to read a credential the caller cannot read (the vault never returns values, stored literals come back "__redacted__", files on the engine host are not readable through the bus):

  • A value kept from the stored entry — a "__redacted__" restored, a ${VAR} template left unchanged — only goes to the same provider and destination (the endpoint_url as scheme, host and port; else AWS, Google, or the R2 account). Moving the endpoint, account or provider needs the value entered again.
  • A secret:// / env:// reference or a credentials_file that a stored bucket uses, in any credential field, only goes in the same field to a destination a stored bucket sends it to (entries that do not parse count too). testBucket counts every stored bucket, and so does saveBucket, except that a bucket moved to a new provider, endpoint or account takes its own access_key_id, secret_access_key and account_id along in the same fields when no other bucket shares them (only an identifier and a signature leave; the access key id follows even as a secret:// reference because it is an identifier sent in clear in every SigV4 request — storing it as a secret keeps it out of the configuration, not out of requests). Its session_token, credentials_json and credentials_file, sent in clear or as bearer credentials, must be entered again, and a value another bucket shares (a Duplicate) needs a new one. An r2 API token always goes to Cloudflare's API, so it follows. A value never changes field. A Duplicate to the same endpoint keeps working.
  • A credentials_file is a file of the engine host: a regular JSON key file of at most 64 KiB (never a FIFO or device), of type service_account or authorized_user, whose token_uri is Google's; one no stored bucket uses only goes to Google. Every failure gives the same message, including a key the auth library or Google refuses.
  • The engine host's ambient credentials — the AWS default chain for an s3 bucket without keys, Google ADC for a gcs bucket without credentials — only go to the provider itself or to an endpoint a stored bucket already uses them with.
  • A request may not introduce a ${VAR} template (it reads the engine's environment, which nothing gates; env://NAME goes through the secrets worker's access rules), nor userinfo in endpoint_url. An r2 account_id must be a 32-hex account id before it becomes a hostname.

credentials stores its values as literals in the configuration, where configuration::get {raw: true} shows them; the secrets vault is the intended store.

storage::listBuckets

{} → { buckets: [...], count, config_id }, sorted by name. Each bucket: name, provider, bucket (the provider-side bucket), region? (s3), endpoint_url? (userinfo stripped), account_id? (r2: the literal or the reference as stored), display?, uses_secrets, direct_downloads (whether a presigned GET works for it: false for gcs without signing credentials or with an endpoint_url override), and for a bucket that could not be built error + error_stage (config | secrets | auth). Stored entries that do not parse are listed too (error_stage: "config").

storage::getBucket

{ name } → { name, bucket }: the entry as stored (${VAR} templates and references unexpanded), sensitive literals and keys that are not fields as "__redacted__" (a save keeps the stored value of either when the placeholder comes back).

storage::saveBucket

{
  "name": "photos",
  "previous_name": "photos",
  "bucket": {
    "provider": "s3", "region": "us-east-1", "bucket": "my-photos",
    "endpoint_url": "https://minio.example.com", "force_path_style": true,
    "access_key_id": "secret://STORAGE_PHOTOS_ACCESS_KEY_ID",
    "secret_access_key": "secret://STORAGE_PHOTOS_SECRET_ACCESS_KEY",
    "display": { "icon": "image", "color": "#3B82F6" }
  },
  "credentials": { "session_token": "…" },
  "create_bucket": true
}

→ { name, saved: true, created_bucket?, error?, error_stage? }.

  • No previous_name creates (a taken name is refused); previous_name == name updates; another previous_name renames (a taken name is refused). A rename of an entry without bucket keeps pointing at the same provider-side bucket.
  • Keys that are not fields of the provider are refused by name, and so are literal sensitive values inside bucket. credentials (access_key_id, secret_access_key, session_token, credentials_json, notifications_api_token) is merged over bucket — trace redaction hides a key named credentials whole. "__redacted__" keeps the stored value.
  • The save is a read-modify-write of the whole raw value through configuration::set: providers, the other buckets and their templates and references are written back untouched. configuration::set has no compare-and-set, but answers the value it replaced: when another writer got in between, the edit is applied again on top of that writer's value (best effort). The answer comes after the runtime reloaded; error / error_stage is the saved bucket's build error (the entry is saved anyway). Refusals are CONFIG_ERROR.
  • Changing a local bucket's bucket points it at another directory under data_dir/buckets/; objects are not moved.
  • Known limit: a writer that rewrites the whole value without re-applying (the console's configuration editor, a hand configuration::set) and read before a save overwrites it. Only a compare-and-set in configuration::set can fix that.
  • create_bucket: true (s3, r2, S3-compatible) creates the provider-side bucket when it does not exist (created_bucket: true; already there → false), sending the region as LocationConstraint except for AWS's us-east-1 and r2. A failure is reported in error with the entry saved. Local buckets always create their directory.

storage::deleteBucket

{ name } → { deleted }. Removes the entry only — never objects, local data or secrets. Removing the last bucket is allowed.

storage::testBucket

{ bucket?, name?, credentials?, timeout_ms? } → { ok, provider, latency_ms, message?, stage?, bucket_missing?, direct_downloads? }.

Builds a throwaway client (one attempt, no retries) with every reference resolved fresh — never from the cache — and lists one object at the root. Failures are data, not errors: stage is config (the entry), secrets (a reference did not resolve), endpoint (DNS, connect, TLS, timeout), auth (rejected keys or signature, no credentials at all), bucket (NoSuchBucket, with bucket_missing: true; another region) or list (anything else). timeout_ms defaults to 10000 (max 30000). Without bucket it tests the stored bucket name; with both, name supplies the "__redacted__" values and unchanged ${VAR} templates — and those values, like a reference a stored bucket already uses, are only sent to that bucket's endpoint. A local bucket checks that its directory can be written.

RPC reference notes

Inline object calls are intentionally limited

storage::putObject and storage::getObject are convenience functions for small configuration fragments, thumbnails, test fixtures, and similar values. They are not streaming APIs: the full object crosses the iii engine as base64 and is held in memory. The decoded body is capped at 10 MiB. For files—even when a file happens to fit below that cap—prefer direct transfer endpoints:

  • upload: storage::presignPost for local multipart POST, or storage::presignUrl with method: "PUT";
  • download: storage::presignUrl with method: "GET".

storage::getObject — ranged reads

offset (default 0) and length (default: to the end) read a byte range of any object; the response adds offset and length (the bytes returned) while size stays the object's total size. The 10 MiB cap applies to the bytes returned, so { offset, length: 2097152 } chunks read a 2 GB video. An offset equal to the size returns an empty body; past it, INVALID_RANGE {offset, size}. S3 and R2 send a Range GET, GCS a ranged download pinned to the generation it read, local storage seeks.

Object keys

Every call that takes a key (putObject, getObject, headObject, deleteObject, presignUrl, presignPost) refuses a key with a . or .. segment and returns INVALID_KEY. Segments are read the way a URL parser reads them: %2e (any case) is a dot, a backslash separates segments like a slash, and a tab, CR or LF is ignored, so ../escape, a/./b, %2e%2e/x and a\..\x are all refused. A path-style S3 endpoint and the presigned URLs of S3, R2 and GCS carry the key in a URL path, where a literal . or .. segment would be resolved to another path or bucket; the GCS presigner also reads %2e, a backslash and tab/CR/LF that way. One rule for every backend keeps a key that works on one from failing on another, at the cost of refusing keys such as C:\dir\..\file that the S3 SDK (which encodes % and \) and the local backend (which hashes keys) would have stored safely. Dots inside a name (a.b, ..a, ...) are fine. listObjects is unaffected: prefix is a query parameter.

An object written before this rule with such a key (possible with the local backend, GCS, S3 virtual-hosted endpoints, and S3 path-style endpoints for keys like %2e%2e/x or a\..\b, which the SDK encoded) still shows in listObjects, but this worker can no longer read or delete it: remove it with the provider's own tools. An empty key keeps its earlier code (CONFIG_ERROR, and INVALID_PRESIGN_PARAMS from presignPost).

storage::presignPost — multipart upload

Returns { url, fields, expires_at }. Add every returned field to a multipart/form-data body before the file part. Native local storage streams the file to disk and can enforce optional max_size_bytes without buffering. Other providers currently return PRESIGN_UNSUPPORTED; use a signed PUT there.

storage::presignUrl — GET-only response-override params

Two optional fields are accepted only when method is "GET". Passing either on a PUT presign returns INVALID_PRESIGN_PARAMS.

Field Type Description
response_content_disposition string (optional) Override Content-Disposition header on the served response (e.g. "attachment; filename=\"report.pdf\"").
response_content_type string (optional) Override Content-Type header on the served response (e.g. "application/pdf").
const { url } = await iii.call({
  function_id: 'storage::presignUrl',
  payload: {
    bucket: 'uploads',
    key: 'reports/q1.pdf',
    method: 'GET',
    expires_in_seconds: 300,
    response_content_disposition: 'attachment; filename="q1.pdf"',
    response_content_type: 'application/pdf',
  },
})

Local signed downloads: Range and HEAD

The native local download route answers HEAD and honours Range, so a video player can seek: Accept-Ranges: bytes on every answer; bytes=a-b, bytes=a- and bytes=-n → 206 Partial Content with Content-Range; a range that starts at or past the end, bytes=-0, or any range of an empty object → 416 with Content-Range: bytes */size. A request for several ranges (bytes=0-1,5-6), another unit, or a malformed or inverted range is answered with the whole object (200), which RFC 9110 allows; If-Range naming another version does the same. Downloads carry X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox, so an HTML object never runs as a page of the listener's origin. The signing key is per worker process: URLs issued before a restart answer 401. presignUrl refuses control characters in response_content_disposition / response_content_type.

Local development & testing

The committed config.yaml declares a single scratch bucket served directly by the worker. Pass it as a seed so the configuration worker picks it up on first boot — zero cloud credentials required.

# In one terminal: start the engine (must include the configuration worker)
iii --config config.yaml

# In another: build & run the worker, seeding config.yaml on first registration
cargo run --release -- --url ws://127.0.0.1:49134 --config ./config.yaml

The worker registers its schema with the configuration worker (seeding config.yaml if no stored value exists), fetches the live config, initializes the native store and optional direct-transfer HTTP listener, and registers storage::putObject, storage::getObject, storage::deleteObject, storage::presignUrl, storage::presignPost, storage::headObject, storage::listBuckets, storage::listObjects, storage::getBucket, storage::saveBucket, storage::deleteBucket and storage::testBucket. Files land under data/storage/ under III_COMPOSE_DIR (configurable via providers.local.data_dir).

Running --manifest prints the registry-publish JSON without touching the engine — useful when testing CI flows:

cargo run -- --manifest | jq .

Tests

cargo test --lib                # unit tests (config, manifest, handlers, object events)
cargo test --test schemas       # schema regression for every `storage::*` RPC
cargo test --test manifest      # `--manifest` subprocess contract
cargo test --test integration   # spec §9 pattern A: spawns engine + worker

tests/integration.rs self-skips when the iii engine is not available on PATH. Native local tests require no external storage dependency. The richer per-provider e2e suite under tests/e2e/ is env-var-gated — see tests/e2e/run-tests.sh for the orchestrator. Its harness covers bucket management, ranged reads and HTTP Range against every selected provider (--providers=local needs nothing; all adds MinIO through docker compose).

Verification before publishing

The full preflight checklist for binary workers (docs/sops/binary-worker.md §11):

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
./target/debug/storage --manifest | jq .

S3 multipart provider contract (opt-in)

This check is for an isolated test bucket, never a production bucket. Set S3_TEST_ENDPOINT, S3_TEST_BUCKET, S3_TEST_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY in your environment (and AWS_SESSION_TOKEN if needed), then run:

cargo test --manifest-path storage/Cargo.toml --test schemas e2e::multipart::multipart_contract -- --ignored --exact

Add --nocapture after --exact to display the conditional probe result.

The endpoint uses path-style addressing. The test creates a unique qa/multipart/<uuid>/ prefix, checks SHA256/COMPOSITE multipart uploads, incorrect checksums, ETags, abort, metadata and ranged reads, and attempts to remove only its own uploads and objects. A missing required variable fails the explicit run. The conditional-completion probe reports whether an existing object was rejected with PreconditionFailed, explicitly reports NotImplemented, or reports that the precondition was ignored. Other errors fail the test. A successful SDK test does not establish browser CORS, exposed ETag, signed Content-Length enforcement, lifecycle cleanup, or provider/version compatibility: verify those separately against the actual product before enabling uploads. Never record credentials or signed URLs in test evidence.

Search the registry

Find workers, kits and executables, or jump to a page.