---
title: "2026-09-02 Factory automation architecture research"
description: "Developed a minimal materialization-reconciler architecture for benchmarking, derived-data generation, and training automation on Argo, vxData, and Kueue."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-02 Factory automation architecture research

## Work Done

Independently compared desired-state reconciliation, asset and materialization
graphs, event-driven automation, queues and schedulers, durable execution, and
experiment tracking for the internal ML automation Factory. The investigation
used the current `apps/factory` implementation in `virdx/mono`, the
infrastructure SOPs for Argo and vxData, the vxData schema model, the inverse
workstream status, and the existing Factory GPU-control audit. No audited
repository or cluster resource was changed.

The recommended core model is a level-triggered materialization reconciler:

`desired materializations - completed materializations - durably claimed materializations = actionable work`

The existing Factory already has the right base shape: immutable rules in code,
pure diffs over observed vxData state, and Argo workflows for in-flight work.
It should evolve rather than be replaced by a second orchestration platform.
Events should only wake or narrow reconciliation; they must not become the
source of truth. A periodic full reconciliation remains the repair path for
missed, duplicated, or reordered events.

The smallest coherent domain model has four concepts:

1. `AssetRevision`: an immutable vxData resource/version or blob identity.
2. `RuleRevision`: an immutable derive, train, or benchmark declaration with
   semantic parameters and pinned executable identity.
3. `Materialization`: the desired application of one rule revision to exact
   input revisions, protected by a deterministic key and unique constraint.
4. `Attempt`: one Argo execution try linked to its materialization, Kueue class,
   terminal status, and optional ClearML task.

Only materializations and attempts need new control-plane persistence. A
Postgres-backed claim transaction or lease closes the current race between
concurrent sync calls and preserves submission and retry history beyond Argo
retention. Argo remains the durable attempt executor, not the claim database.
Completion is proven by a valid, idempotently published vxData output, not by an
Argo `Succeeded` phase alone.

The recommended thin Argo-native composition is:

- optional Hera for typed Python authoring of Argo workflows;
- Argo Events as a low-latency reconciliation wakeup;
- Argo Workflows for execution, retries, fan-out, and timeouts;
- Kueue for quota admission, fairness, priority, and preemption;
- vxData for immutable assets, materialization completion, and provenance;
- Postgres for materialization claims and attempt history;
- ClearML as experiment telemetry and an index projection, not canonical
  lineage or completion truth.

KFP, Dagster, and Metaflow should not become Factory core by default. Each
introduces a second control plane and overlaps current Argo, vxData, or ClearML
responsibilities. Adopt one only through an intentional platform decision that
accepts its ownership, migration, and operational costs.

Workload-specific recommendations were:

- Derived data uses one materialization per natural atomic case and records
  exact `derived_from` input versions.
- Benchmarks key results by model revision, frozen cohort-manifest revision,
  evaluator revision, and metric policy. Live queries must not silently define
  reproducibility-critical cohorts.
- Training starts from an explicit immutable dataset snapshot or
  `TrainingRequest`, not every raw-data event. Budget and approval policy make
  training desired; Kueue decides when it runs. Produced models then make
  compatible benchmark materializations desired.

## Lessons Learned: Pitfalls

The current Factory's `desired - done - Argo in-flight` model is stronger than a
simple cron-triggered job launcher, so a generic greenfield orchestration design
would have duplicated working behavior. Reading the implementation before
selecting products was necessary.

Argo provides durable execution but not durable domain intent. Terminal workflow
retention, submit timeouts, concurrent reconcilers, and retry backoff cannot be
modeled safely by treating Argo as the only work ledger.

Events, Kueue admission, and ClearML runs can all look like automation state, but
none establishes whether a semantic output should exist and is complete. Mixing
those roles would create split-brain truth.

The legacy `~/.virdx/knowledge` checkout was divergent while the required
`~/.agents/knowledge` checkout was current. Using the latter avoided altering or
rebasing unknown local work.

## Lessons Learned: Improvements

Add a durable Factory architecture guide under the infrastructure workstream
after the design is accepted. It should record the materialization identity
contract, semantic versus execution parameters, output publication rules,
Postgres claim transaction, and the ownership boundary among vxData, Argo,
Kueue, ClearML, and Argo Events.

Document whether vxData exposes immutable resource-version identifiers and a
transactional outbox. Those two details decide the exact materialization key and
safe event-wakeup design.

Add an indexed lookup for materialization key, producer revision, and exact input
revision before the rule catalog grows. Repeated full-payload scans are suitable
for the current small rule set but are not the long-term graph query model.

Record explicit adoption criteria for KFP, Dagster, Metaflow, or Temporal so a
new tool is added only when a required capability cannot be supplied by the
thin Argo-native composition.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-02-infrastructure-factory-automation-architecture/index.mdx