Skip to content
VIRDX

VxData API: explicit bundle references

Bundles use the ordinary POST /s3/presign/upload-dir allocation contract:

{
  "group": "example",
  "relpaths": ["shard-000000.tar", "manifest.parquet"]
}

The optional bucket field selects a configured S3 bucket. The response contains root and presigned items. All directory allocations validate unique safe relative paths, allowing ordinary names containing colons or backslashes. SDK directory uploads send POSIX separators; portable bundle paths retain stricter validation in vxs3. The API has no bundle-specific allocation metadata or completion acknowledgement. Vxs3 verifies shards and publishes/verifies the manifest last, then returns the credential-free manifest URI and hash for catalog registration.

SDK downloads request with_sizes=true on /s3/presign/batch; this stats each object. The API retains the optional flag for other consumers.

Bundle is the dedicated resource type for a published bundle, identified as bundle/<uuid4>. Its manifest_url and manifest_sha256 pin the exact stored vxs3 manifest (manifest.parquet, one row per file or directory); manifest_meta copies the parquet footer. Optional records (records_url, records_sha256, records_schema, records_meta, set together) describe the uploader’s own parquet table whose path_* columns name bundle files. title is required. Resource creation, reads, queries and updates use the existing authorization and versioning rules; principals that register bundles need permission to write Bundle resources. GenericFile describes one physical object and keeps its existing URI and checksum fields.

The API records the publisher’s completed references; it does not download shards or independently verify publication as part of registration or a resource query. The SDK’s client.bundles.upload(...) stores records, publishes and verifies the bundle, then registers and returns the Bundle resource. Registration failure does not roll back published bytes; the SDK retains the create request in BundleRegistrationError.resource for recovery without another upload.

Apply migration b7e4c2a9d5f1 before deploying this API version. It renames the FileBundle table and resource type to Bundle, flattens the former manifest reference into manifest_url/manifest_sha256, adds the title and records columns, and titles existing rows with their resource identifier. Existing rows keep pointing at their manifest.json.gz; vxs3 still reads that v3 manifest, so no data migration is needed. Migration 926ab0338a7b (the original FileBundle table) precedes it. Use the matching SDK’s client.bundles namespace for publication and consumption. Raw /s3/list and ordinary directory uploads retain their physical-object semantics. Resolving legacy worker metadata and transparent logical listing/downloads are separate work.

crud

Navigation

Type to search…

↑↓ navigate↵ selectEsc close