2026-09-02 Factory automation architecture research

Work Done

Independently compared desired-state reconciliation, asset and materialization graphs, event-driven automation, queues and schedulers, durable execution, and experiment tracking for the internal ML automation Factory. The investigation used the current apps/factory implementation in virdx/mono, the infrastructure SOPs for Argo and vxData, the vxData schema model, the inverse workstream status, and the existing Factory GPU-control audit. No audited repository or cluster resource was changed.

The recommended core model is a level-triggered materialization reconciler:

desired materializations - completed materializations - durably claimed materializations = actionable work

The existing Factory already has the right base shape: immutable rules in code, pure diffs over observed vxData state, and Argo workflows for in-flight work. It should evolve rather than be replaced by a second orchestration platform. Events should only wake or narrow reconciliation; they must not become the source of truth. A periodic full reconciliation remains the repair path for missed, duplicated, or reordered events.

The smallest coherent domain model has four concepts:

1. AssetRevision: an immutable vxData resource/version or blob identity.
2. RuleRevision: an immutable derive, train, or benchmark declaration with semantic parameters and pinned executable identity.
3. Materialization: the desired application of one rule revision to exact input revisions, protected by a deterministic key and unique constraint.
4. Attempt: one Argo execution try linked to its materialization, Kueue class, terminal status, and optional ClearML task.

Only materializations and attempts need new control-plane persistence. A Postgres-backed claim transaction or lease closes the current race between concurrent sync calls and preserves submission and retry history beyond Argo retention. Argo remains the durable attempt executor, not the claim database. Completion is proven by a valid, idempotently published vxData output, not by an Argo Succeeded phase alone.

The recommended thin Argo-native composition is:

• optional Hera for typed Python authoring of Argo workflows;
• Argo Events as a low-latency reconciliation wakeup;
• Argo Workflows for execution, retries, fan-out, and timeouts;
• Kueue for quota admission, fairness, priority, and preemption;
• vxData for immutable assets, materialization completion, and provenance;
• Postgres for materialization claims and attempt history;
• ClearML as experiment telemetry and an index projection, not canonical lineage or completion truth.

KFP, Dagster, and Metaflow should not become Factory core by default. Each introduces a second control plane and overlaps current Argo, vxData, or ClearML responsibilities. Adopt one only through an intentional platform decision that accepts its ownership, migration, and operational costs.

Workload-specific recommendations were:

• Derived data uses one materialization per natural atomic case and records exact derived_from input versions.
• Benchmarks key results by model revision, frozen cohort-manifest revision, evaluator revision, and metric policy. Live queries must not silently define reproducibility-critical cohorts.
• Training starts from an explicit immutable dataset snapshot or TrainingRequest, not every raw-data event. Budget and approval policy make training desired; Kueue decides when it runs. Produced models then make compatible benchmark materializations desired.

Lessons Learned: Pitfalls

The current Factory’s desired - done - Argo in-flight model is stronger than a simple cron-triggered job launcher, so a generic greenfield orchestration design would have duplicated working behavior. Reading the implementation before selecting products was necessary.

Argo provides durable execution but not durable domain intent. Terminal workflow retention, submit timeouts, concurrent reconcilers, and retry backoff cannot be modeled safely by treating Argo as the only work ledger.

Events, Kueue admission, and ClearML runs can all look like automation state, but none establishes whether a semantic output should exist and is complete. Mixing those roles would create split-brain truth.

The legacy ~/.virdx/knowledge checkout was divergent while the required ~/.agents/knowledge checkout was current. Using the latter avoided altering or rebasing unknown local work.

Lessons Learned: Improvements

Add a durable Factory architecture guide under the infrastructure workstream after the design is accepted. It should record the materialization identity contract, semantic versus execution parameters, output publication rules, Postgres claim transaction, and the ownership boundary among vxData, Argo, Kueue, ClearML, and Argo Events.

Document whether vxData exposes immutable resource-version identifiers and a transactional outbox. Those two details decide the exact materialization key and safe event-wakeup design.

Add an indexed lookup for materialization key, producer revision, and exact input revision before the rule catalog grows. Repeated full-payload scans are suitable for the current small rule set but are not the long-term graph query model.

Record explicit adoption criteria for KFP, Dagster, Metaflow, or Temporal so a new tool is added only when a required capability cannot be supplied by the thin Argo-native composition.