Skip to content

2026-09-16 vxData remote random access reader

Added one SDK method for safe range-backed reads and opened mono PR 403.

Work Done

Opened mono PR 403, commit d0bdf643 on fk-vxdata-open-remote. Adds storage.open_remote(url), a read-only seekable RawIOBase backed by existing signed-URL/httpx transport. Fixed 256KiB aligned blocks, 8-block/2MiB cache, strong ETag/If-Match consistency, range/body validation and bounded retries/URL refresh. Existing storage.open and bulk downloads unchanged. No new dependencies or manual version bump (Nx Release owns package versions).

Four files changed: storage namespace, private range reader, focused tests, SDK README. 96 SDK tests, lint/typecheck/format, module boundaries and conda build/import checks passed. New method read seven TIFF pyramid levels plus resolution from the failing 779,326,663-byte Gleason input using 8 GETs / 2,069,703 bytes in a single local-client 2.17s measurement. No publication, Histo integration or training deployment performed.

Pitfalls

Generated conda output can be recursively collected by broad pytest invocations; remove build output or target tests/. The reader is not thread-safe and the Client must remain open. It has no local filename and read() without a size intentionally allocates the remainder. Mutation is detected on future network reads; cached bytes remain a snapshot.

Improvements

Use open_remote for sparse metadata access and open/download for materialization. This avoids public signed-URL plumbing and does not alter the fixed bulk-transfer policy. Validate additional layouts and cluster-scale performance before replacing Histo metadata reads.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close