Inspected the vxdata-production API pod with read-only kubectl commands.
Found six /s3/presign/batch HTTP 500 responses between 09:09 and 09:14 UTC.
Confirmed every failure was an uncaught MinIO S3Error with NoSuchKey while with_sizes=True caused stat_object to run on a domain-representation directory prefix.
Traced the malformed URL to diffsim/apps/gen4ve2e/generate_vxdata.py: removing the full "/gleason_seg.zarr/" suffix also removes the directory prefix’s trailing slash, so the SDK treats it as a single object rather than calling /s3/list.
Lessons Learned: Pitfalls
The infrastructure SOPs were useful for locating the production namespace, but no existing entry described the SDK’s trailing-slash distinction for downloads.
A batch endpoint failure can be misleading: one invalid item aborts the batch and surfaces as HTTP 500 even when batch presigning itself is healthy.
Lessons Learned: Improvements
Document that vxd.storage.download treats only S3 URLs ending in / as prefixes. Prefix URLs without / are presigned as objects and size lookup can raise NoSuchKey.
The API should convert missing keys during presign into a clear 4xx response or a per-item batch error rather than an uncaught 500.
