Greenfield Pipeline Design

1 · Problem and requirements

Every company that moves goods runs on two kinds of information. The record: an order was placed, an invoice issued, a container booked. And the traffic: the emails, PDFs, WhatsApp messages, phone calls and portal screens through which the record gets made and kept current. ERPs hold the record. The traffic is read by people, one message at a time, and typed into the record by hand. Aradus does the traffic. It reads what arrives, decides what it means, does the next thing or asks a person, and remembers how it decided so it does not ask twice. The ERP, or the spreadsheet where there is no ERP, stays the record and is written last.

Four layers, and the design is those four done properly. Edge: the doors in and out (email, WhatsApp, voice, uploads, provider feeds). Reading: turning pixels and text into candidate facts with a confidence and the evidence behind them. Deciding: a policy or a person turns a candidate into a fact and acts on it. Memory: what was decided, on what evidence, and the rule that came out of it. The last layer is the one no ERP has and the reason a customer stays.

Four kinds of record stay separate throughout. A delivery is how something arrived and from whom. An artifact is the bytes, stored once. A document is one business document inside an artifact, and it exists only once classification has decided what it is. A business object is the shipment, order, bill or specification version the documents are about. Most of the current system's incidents come from collapsing two of these into one row.

Two shapes of tenant use the same design. A trading company with a small ERP, or none, where Aradus may end up as the record of orders and bills. And a manufacturer already on an ERP such as Odoo, where the ERP owns every commercial document and Aradus writes only a suggestion, a flag, or a value a person has approved. The RVK Factory assessment is the second shape at its extreme: quotations from bills of quantities, engineering drawings and voice notes as input, and a rollout rule that humans approve everything. objects.authority (3.5) is the switch between the two shapes; the tables and workflows do not change.

Functional requirements

• Edge. Ingest through upload, email, WhatsApp, chat, API, ERP attachments and provider feeds, message bodies and attachments alike, voice notes and engineering drawings included; store one copy per file and record every arrival with sender, channel and time. Send on the same channels through idempotent dispatches, with WhatsApp's template rule respected.
• Reading. Classify, split, extract, normalize and validate about 25 document types, nine of which create or update business records; read statements in message bodies as assertions about existing objects; transcribe voice notes and keep the audio as the record; read drawings and specification sheets region by region, quantities from the schedule table rather than counted symbols; match names and codes to parties, products, ports, carriers and vessels through scoped aliases; link documents and messages to shipments and orders, and send genuine ambiguity to review.
• Deciding. Version supplier bills with a materiality gate and a fixed tie-break; confirm artwork per order line before a supplier PO; track shipments by bill of lading with provider precedence and a vessel fallback; three-way match; freight requests, quotes and bookings; generate packing lists and commercial invoices from a loading list; forecast demand and supplier lead time from history and propose the stock level or date; suggest a product code while a person types, from that customer's history; answer or route what people send; notify on state changes and in digests; sync with Odoo and Business Central without changing the accounting behaviour customers rely on.
• Memory. Every accepted value traceable to its proposal, evidence, policy and decision with the actor recorded; every correction a labeled example; every intervention offers to become a rule; per-tenant and per-counterparty policy for thresholds, contacts, modes and templates; the usage and cost of every model call, failed ones included.

Non-functional requirements

• Multi-tenant with row-level security: a bug in one tenant's query cannot read another's rows.
• At-least-once delivery everywhere and every effect idempotent; a billable model call is not repeated on redelivery; a retried send goes out once.
• Conversations and chases survive restarts and deploys: a reminder due in two days fires in two days.
• Accuracy measurable per stage, tenant and document type, and attributable to a model, prompt or dataset change.
• Autonomy configurable by field, counterparty and document type, gated on calibrated confidence rather than model self-assessment, with a mode as well as a threshold.
• Evidence retained per tenant policy with legal hold; a customs dispute runs years.
• Onboarding a new counterparty format or a new tenant rule costs configuration, not engineering.
• A suggestion asked for from an ERP screen answers in under a second from a cache; no model call sits in that request path.
• The model provider is a per-tenant policy, so a tenant that needs in-country processing gets a self-hosted transcription or language model without a code change.

Scale honesty

This is a correctness problem rather than a throughput one. Production creates on the order of one shipment a day and tens of orders a month per tenant; at a hundred times that, one Postgres primary handles every table here without partitioning. The scarce resources are model spend, human review minutes and correctness under retries, and the design optimizes for those.

2 · Architecture

Five components, and the reason each earns its place:

• One modular Go codebase deployed as API and worker pools. No microservices: the transactional boundaries around documents, obligations, shipments and reviews matter more than deployment independence at this scale.
• Self-hosted Temporal on the existing Kubernetes cluster handles execution: leases, retries, timers, heartbeats, fan-out and fan-in, review waits, conversation signals, cancellation, and continuity across deployments. Postgres stays the only business truth; the UI and API read run status from Postgres projections, not from Temporal. Self-hosting is a data-residency decision and carries an operations cost: four services plus persistence, and someone on call who has run it.
• Control flow is code and tenant variation is policy. Workflows are written and versioned like code; every value a tenant changes (recipients, reminder intervals, thresholds, modes) is a policy row read at each step. There is no operator-editable graph builder; the operator surface is a rulebook. Conversations with people are workflows too, one per contact and channel, so headless and chat-driven steps share one code path (section 6).
• A Python media and model gateway, deliberately shrunk: PDF and image rasterization (PyMuPDF has no Go equal), provider adapters and their quirks, and the the process for testing and improving extraction prompts and examples (few-shot assembly, future future automated prompt optimizers). It is stateless, has no database credentials, and schemas are defined once in JSON Schema and used to generate matching Go and Python types, so the the Go and Python schema definitions becoming inconsistent the current system suffers cannot recur. If the PDF tooling gap ever closes, this component can move into the Go application; nothing else depends on its language.
• PostgreSQL, one primary: Authoritative stored data, row-level security provides a second enforcement layer between customers' data, and an append-only domain event log written by activities alongside each fact: a record for audit and analytics rather than a transport (section 6, consistency without an outbox).
• S3: the original files (written once, then read-only) plus derived files such as page images and crops, which are regenerable copies rather than authoritative source data.

3 · Data model

Three rules decide every table on this page:

1. A table exists only for a thing we originate or adjudicate that needs a database-enforced constraint: uniqueness, idempotency, append-only history, or tenancy.
2. A business object the ERP may own is one row in objects, whatever its kind, with the snapshot stored as JSON validated by a per-kind JSON Schema file, and generated columns for the fields everything joins on. For a tenant with no ERP the same row is the record, authority aradus.
3. A fact about an object that is not in the snapshot is a facts row under a registered attribute, reduced into the current_facts materialized view by the attribute's registered reducer.

erDiagram
    conversations ||--o{ deliveries : "inbound - one thread per tenant, channel, contact"
    conversations ||--o{ dispatches : "outbound - the mirror of deliveries"
    deliveries ||--o{ delivery_artifacts : "body and attachments alike"
    delivery_artifacts }o--|| artifacts : "many arrivals, one stored file"
    artifacts ||--o{ artifacts : "parent_artifact_id - renditions are artifacts with a kind"
    artifacts ||--o{ documents : "artifact_id"
    deliveries ||--o{ documents : "delivery_id - the producing arrival, same file enforced"
    pipeline_proposals |o--o{ documents : "created_from_proposal_id UNIQUE - born typed"
    documents |o--|| objects : "every documents row has a matching objects row of kind document"
    artifacts |o--o{ runs : "file-level runs, subject is the document's object row before it is born typed"
    objects |o--o{ runs : "subject_id - per-object runs, including checks over a set"
    runs ||--o{ pipeline_proposals : "run_id"
    runs ||--o{ model_invocations : "reserved before dispatch - billing truth"
    fitted_models ||--o{ pipeline_proposals : "produced_by.calibrator_version"
    pipeline_proposals ||--o{ proposal_decisions : "append-only verdicts - latest wins"
    asks ||--o{ ask_items : "ask_id"
    proposal_decisions |o--o{ ask_items : "ask_item_id when a person decided"
    specifications ||--o{ specification_versions : "specification_id"
    proposal_decisions |o--o{ aliases : "a correction writes a scoped alias"
    proposal_decisions |o--o{ object_links : "an accepted link creates the edge"
    objects ||--o{ object_links : "from_id"
    objects ||--o{ object_links : "to_id - objects for every kind but two: contact_of and pins_specification target contacts and specification_versions instead"
    objects ||--o{ objects : "parent_id - order_line under order, obligation_line under obligation_revision, location under partner..."
    runs ||--o{ materializations : "accepted set digest - idempotent persist"
    materializations }o--|| objects : "the object it created or updated"
    objects ||--o{ facts : "subject_id"
    attribute_definitions ||--o{ facts : "attribute_definition_id"
    objects ||--o{ contacts : "partner_id - a contact belongs to a partner object"
    contacts ||--o{ contact_channels : "contact_id"
    organizations ||--o{ policies : "the config cascade"

    deliveries {
        varchar id PK
        varchar organization_id FK
        intake_channel channel
        varchar contact_id FK "sender identity"
        varchar conversation_id FK "thread"
        varchar idempotency_key UK
    }
    artifacts {
        varchar id PK
        varchar organization_id FK
        varchar parent_artifact_id FK "NULL - set on a rendition"
        text content_hash "UNIQUE per org"
        text storage_tier "cost"
        timestamptz retain_until "obligation"
    }
    documents {
        varchar id PK
        varchar artifact_id FK "NOT NULL"
        varchar delivery_id FK
        varchar created_from_proposal_id FK "UNIQUE - retry-safe identity"
        integer generation "resegment supersedes the set"
        text artifact_part "pages or sheet"
        varchar document_type "NOT NULL - born typed"
    }
    runs {
        varchar id PK
        varchar subject_id FK "an objects row"
        run_purpose purpose "pipeline reprocess shadow check forecast"
        run_status status "running waiting completed failed superseded"
        varchar policy_snapshot_id
    }
    pipeline_proposals {
        varchar id PK
        varchar run_id FK
        proposal_type proposal_type "classification field assertion refers_to entity_match link revision check"
        varchar subject
        smallint pass "UNIQUE run+type+subject+pass"
        jsonb proposed
        jsonb source_ref "page bbox - what review highlights"
        numeric confidence "calibrated"
        jsonb produced_by "model prompt schema policy versions"
    }
    proposal_decisions {
        varchar id PK
        varchar proposal_id FK
        decision decision "accept correct reject defer"
        jsonb decided_value "corrections land here"
        varchar actor_type "policy user counterparty import process"
        varchar supersedes_decision_id FK "re-decisions append"
    }
    materializations {
        varchar id PK
        varchar run_id FK
        varchar object_id FK "NULL when the target is a fact"
        text proposal_set_digest "UNIQUE(run, target_type, target_id)"
    }
    objects {
        varchar id PK
        varchar organization_id FK
        varchar kind "order shipment obligation partner product ... - no fixed enum, a schema file per kind"
        varchar parent_id FK "NULL"
        varchar authority "aradus or erp"
        jsonb snapshot "validated against the kind's JSON Schema"
    }
    object_links {
        varchar id PK
        varchar from_id FK
        varchar to_id FK "objects, with the two named exceptions"
        varchar kind "allocation reserves finalizes ... evidence source"
        jsonb payload
    }
    facts {
        varchar id PK
        varchar subject_id FK
        varchar attribute_definition_id FK
        jsonb value
        varchar modality "contracted expected actual"
        text payload_hash "UNIQUE per org - a repeated report is one fact"
    }

Overview of the pipeline core and its seams; the DDL blocks below are authoritative for columns and constraints, and every business object – orders, freight, shipments, obligations, partners, products – is drawn once, in section 3.4.

Conventions: every tenant-owned table carries organization_id with composite foreign keys (foreign keys containing both the organization ID and record ID) and RLS (postgreSQL row-level security (RLS), which restricts each query to the customer's rows); global reference data (countries, currencies, ports, carriers, vessels) does not. created_at/updated_at everywhere, omitted below. Types are abbreviated. Critical database constraints that enforce the design are spelled out because they are the design.

What the database enforces and what code enforces are two different lists, kept separate on purpose. A generated column with a constraint guards money and evidence: the number, status and total columns on objects, the one-live-head partial unique index on an obligation revision, the payload hash that makes a repeated fact a no-op. Everything else, the long tail of what a kind's snapshot holds, is schema-validated JSON rather than a column: every kind has a JSON Schema file with fixtures checked in CI, every reducer has a golden set of inputs and expected outputs, and a test walks every objects row of every kind in a production copy through its current schema file and fails on the first row that does not validate.

3.1 Intake and evidence

deliveries                          -- every arrival, repeat provenance kept; the UNIT of context for reading
  id PK, organization_id, channel (uploaded|email|whatsapp|chat|api|generated),
  actor_type (user|system|counterparty), actor_id NULL,
  contact_id -> contacts NULL,        -- the sender: strongest identity signal, captured here
  conversation_id -> conversations NULL,   -- the (tenant, channel, contact) thread this belongs to
  message_id NULL, in_reply_to NULL, thread_key NULL,   -- provider threading; quoted text is stripped at normalize
  forwarded bool, caption_text NULL,  -- WhatsApp exposes a forwarded flag and withholds the original sender
  requested_context jsonb NULL,       -- shipment hint, forced type, reprocess intent
  idempotency_key, received_at
  UNIQUE (organization_id, channel, idempotency_key)

delivery_artifacts                  -- body and attachments alike; a text-only message has exactly one row
  delivery_id, artifact_id -> artifacts, role (body|attachment|inline), ordinal, filename NULL
  UNIQUE (delivery_id, artifact_id)

conversations                       -- one per (tenant, channel, contact); the identity of the Temporal entity workflow
  id PK, organization_id, channel, contact_id -> contacts,
  temporal_workflow_id, last_inbound_at NULL, last_outbound_at NULL
  UNIQUE (organization_id, channel, contact_id)

dispatches                          -- every OUTBOUND message; the mirror of deliveries
  id PK, organization_id, conversation_id -> conversations, channel,
  kind (pack|reminder|question|receipt|reply|document),
  ask_item_ids uuid[] NULL,           -- the open questions this message poses; a reply that answers them is a decision
  template_key NULL, body_ref, artifact_ids uuid[],
  idempotency_key, status (queued|sent|delivered|read|failed), provider_message_id NULL,
  sent_at NULL, failed_reason NULL
  UNIQUE (organization_id, idempotency_key)   -- a retried activity sends once

artifacts                           -- the received file, stored once, addressed by its hash, read-only after write
  id PK, organization_id, content_hash, byte_size, mime_detected,
  parent_artifact_id -> artifacts NULL,   -- set on a rendition: a derived file rather than a fresh arrival
  kind (source|page_png|text_layer|region_crop|sheet_file|transcript|vector_text),
  -- source: what arrived. transcript: word-timed text from an audio artifact; vector_text: text and geometry read from a DXF without a model
  storage_key, storage_tier (hot|cold|archive),        -- cost, ops-owned
  retain_until NULL, legal_hold bool                   -- obligation, tenant-owned; renditions inherit the source's retention
  UNIQUE (organization_id, content_hash)

documents                           -- the DECIDED semantic unit; zero self-references
  id PK, organization_id, artifact_id, delivery_id,    -- which arrival produced it
  created_from_proposal_id UNIQUE,    -- the decided classification proposal: a retry cannot create a duplicate document
  generation,                         -- reclassification supersedes the prior set; one current generation per artifact
  artifact_part NULL,                 -- page range or sheet name; NULL = whole artifact
  document_type NOT NULL,             -- born typed: a document exists only once classification has decided
  extraction_variant NULL, current_run_id NULL, deleted_at NULL
  FK (organization_id, document_type) -> document_types
  FK (delivery_id, artifact_id) -> delivery_artifacts (delivery_id, artifact_id)   -- the producing delivery must contain this file
  -- every documents row has a matching objects row of kind document (3.4); a version chain is object_links of kind revises
  -- between those object rows, and the current version of anything with money on it is the obligation_revision that cites the document

What this buys, versus the current model: documents loses all four self-referencing foreign keys (columns that point to another row in the same table). Documents split from a larger file are rows with an artifact_part; a merged PDF is an artifact, and no placeholder document row for the unsplit file exists for it. Duplicate handling is a constraint plus a delivery row instead of a unused column and two inconsistent code paths. Retention and tiering are separate fields with separate owners. Renditions are no longer their own table: a page image or a transcript is an artifacts row with parent_artifact_id set and a kind other than source, so the hash, retention and storage-tier columns that already exist do double duty instead of being reimplemented once per derived-file table.

A message is a delivery whose body is an artifact like any attachment. A WhatsApp text with no file has one delivery_artifacts row of role body; an email with two PDFs has three. Classification runs per artifact but sees every artifact in the delivery: the body's "revised invoice attached, disregard the previous one" is context for the attachment's run, and a body that carries no fact of its own is classified as noise and produces no document. Threading fields are captured at intake so a reply can be matched to its conversation and to the question it answers; quoted text is stripped at normalize so a forwarded message does not re-assert facts already recorded.

3.2 Processing: runs, proposals, decisions

runs                                -- thin projection; Temporal owns execution
  id PK, organization_id, subject_id -> objects,
  purpose (pipeline|reprocess|shadow|check|forecast),   -- check: a run over a set of linked objects (3.4); forecast: a fitted_models run
  status (running|waiting|completed|failed|superseded),
  code_version, policy_snapshot_id, temporal_workflow_id
  UNIQUE (subject_id) WHERE status IN ('running','waiting')   -- one active run per subject
  -- a file-level (classify/split) run's subject is the objects row of kind document created at intake; the
  -- documents row itself is inserted only once classification decides the type, so documents stays born typed

pipeline_proposals                     -- IMMUTABLE machine observations
  id PK, organization_id, run_id, document_id NULL, stage,
  proposal_type (classification|segment|field|entity_match|entity_create|link|refers_to|assertion|revision|check),
  -- refers_to: which object a MESSAGE is about, with its own threshold. assertion: a statement about an
  -- existing object (attribute, value, modality, effective_at) that materializes a fact (3.4). check: a deterministic validator result
  subject, pass smallint,             -- pass 2 = targeted high-fidelity re-read
  proposed jsonb, normalized jsonb,
  source_ref jsonb,                   -- page+bbox / sheet+cell / text span (artifact, char_start, char_end) / audio span (artifact, ms_start, ms_end): what the review UI highlights or plays
  features jsonb,                     -- candidate set, agreement, validator results
  confidence numeric NULL,            -- calibrated, not self-reported
  produced_by jsonb                   -- model, prompt, schema, template, policy versions; calibrator_version; few_shot_examples: the decision ids the prompt carried, so an extraction replays with what it saw
  UNIQUE (run_id, proposal_type, subject, pass)

proposal_decisions                  -- APPEND-ONLY; latest per proposal wins
  id PK, organization_id, proposal_id,
  decision (accept|correct|reject|defer), decided_value jsonb NULL,
  actor_type (policy|user|counterparty|import|process), actor_id, rationale NULL,   -- counterparty: a customer's reply, arriving as a delivery, decides an ask item
  ask_item_id NULL, supersedes_decision_id NULL, decided_at,
  sample_reason (routed|audit|gold|counterparty),   -- why a person saw it: routed below threshold, pulled at random from auto-accepted work, a gold case, or a customer's reply
  eval_holdout bool                   -- held out of all training sets; the calibrator and the few-shot pool do not read it

asks                                -- one per set of questions put to someone; typed handlers per kind
  id PK, organization_id,
  kind (field|entity|association|version|erp_conflict|approval|artwork|po_acknowledgement|order_readback|substitution|proforma|onboarding_docs),
  audience (user|counterparty), assignee_user_id NULL, counterparty_partner_id NULL, contact_role NULL,
  subject_type, subject_id,           -- the order, PO, document or partner the ask is about
  policy_id -> policies,              -- chase interval, cap, escalation, what counts as an answer, which step it blocks
  state (open|complete|abandoned), created_at

ask_items                           -- immutable: one row per thing asked about
  id PK, ask_id, organization_id,
  item_ref_type (order_line|specification_version|document_requirement|proposal), item_ref_id,
  asked jsonb, asked_schema_version, dispatch_id -> dispatches NULL   -- the send that carried this item, if any
  UNIQUE (ask_id, item_ref_type, item_ref_id)

materializations                    -- idempotent persist record
  id PK, organization_id, run_id, document_id, object_id -> objects NULL,   -- NULL when the accepted set wrote a fact rather than an object
  target_type (object|fact), target_id, proposal_set_digest
  UNIQUE (document_id, target_type, target_id)   -- the instance is part of the key: one message may update two shipments

model_invocations                   -- the billing source of truth
  id PK (deterministic, caller-supplied), organization_id, run_id NULL, purpose,
  model, prompt_version, schema_version, request_fingerprint,
  status (reserved|completed|failed|ambiguous|reconciled),
  provider_generation_id NULL, tokens_in, tokens_out, tokens_cached,
  provider_cost NULL, latency_ms, reserved_at, completed_at NULL

fitted_models                       -- a fitted mapping or model, immutable per version; what produced_by.calibrator_version points at, or what a forecast fact traces to
  id PK, organization_id NULL,         -- NULL = platform level; a tenant row shrinks toward the platform row when its data is thin
  kind (calibration|demand|lead_time),
  scope (document_type NULL, field_path NULL, layout NULL, counterparty_id NULL, subject_type NULL, subject_id NULL),
  -- a calibrator scopes by document shape; a demand or lead_time fit scopes by subject (product, supplier_category, product_channel)
  method, mapping jsonb,                -- isotonic bins or logistic weights over pipeline_proposals.features; or a quantile model's params
  trained_on daterange, n_examples, n_audit, n_gold,
  backtest jsonb,                       -- calibration: calibration error and the auto-accept error rate at each threshold. forecast: pinball loss, p10-p90 coverage and bias per horizon
  version, created_at
  UNIQUE (organization_id, kind, scope, version)

This trio is the core of the design. Proposals do not change: they are what the machine said, and they are the training labels. Decisions only add new rows without changing earlier ones: A later decision replacing an earlier one, a second reviewer, a reprocess each add a row and replace while retaining the earlier version, so history survives. Domain rows are written only from accepted proposals, recorded in materializations with an input digest, so a redelivered message matches the existing record and makes no change. Corrections carry both the untouched proposal and the decided value, which is what makes each one a labeled example. Accuracy per stage is a GROUP BY over these tables; the eval corpus grows as a side effect of operating the product. A retry within one run no longer has its own attempt ledger: pipeline_proposals' own UNIQUE (run_id, proposal_type, subject, pass) and the materialization digest are what make replaying a run safe, so the same guarantee costs one fewer table.

A forecast is an ordinary use of this trio. Fitting one is a run of purpose forecast over a product or supplier object; its output is an assertion proposal like any other, decided by policy or a person, and materialized as a facts row (subject the product or supplier, attribute forecast_p50 and its band, modality expected, source_kind model) and, where policy allows, an erp_commands push to the ERP's reordering rule or the PO's planned date. The fitted_models row is what makes the forecast able to show, next to the number, how it did on last year's data. Cold-start tenants get the schema and no model: capture from day one, switch a kind on once its backtest is good enough to show a reviewer.

3.3 Identity

organizations                       -- the TENANT: the account that logs in, has members, roles, a plan, integrations. Unchanged.
  id PK, name, slug, ...                -- counterparty rows leave this table; an organization without members is a bug

contacts       id PK, organization_id, partner_id -> objects,   -- the partner object this person's home is; kind partner enforced by the writer
                name, title NULL
contact_channels                    -- a person has N addresses; a conversation resolves an address to one contact
  id PK, organization_id, contact_id -> contacts, channel (email|whatsapp|phone),
  address, verified_at NULL, opt_in_at NULL
  UNIQUE (organization_id, channel, address)

identifiers                         -- deterministic tier: matched before anything fuzzy
  id PK, organization_id NULL,        -- NULL = identifier of a global entity
  entity_type, entity_id, id_type (tax_id|sku|unlocode|scac|imo|iata_prefix),
  value
  UNIQUE (COALESCE(organization_id,''), id_type, value)

aliases                             -- learned vocabulary, scoped, provenance-bearing
  id PK, organization_id, entity_type, entity_id,
  counterparty_id -> objects NULL,    -- scoped wherever the counterparty (a partner object) is known
  alias_text, normalized_text,
  source (correction|operator|import|promoted),
  created_from_decision_id NULL, effective daterange NULL
  UNIQUE (organization_id, entity_type, normalized_text, COALESCE(counterparty_id,''))
  -- one meaning per scoped string; a second entity behind the same string is a review task

specifications                     -- a customer-specific specification of a product that the customer approves as a document
  id PK, organization_id, customer_partner_id -> objects, product_id -> objects,
  kind (artwork|label|formula|packaging|quality_standard), name, current_version_id -> specification_versions NULL
  UNIQUE (organization_id, customer_partner_id, product_id, kind, name)

specification_versions             -- append-only; one row per file ever received
  id PK, organization_id, specification_id -> specifications, version_no,
  artifact_id -> artifacts,            -- the file, content-addressed
  status (candidate|current|superseded|rejected),
  created_from_proposal_id -> pipeline_proposals UNIQUE,   -- the decided refers_to proposal; a retry cannot create a duplicate
  received_via_delivery_id -> deliveries, promoted_by_decision_id NULL, created_at
  UNIQUE (specification_id, version_no)
  UNIQUE (specification_id) WHERE status = 'current'
  -- an attribute that needs no approval, an origin, a class, a grower's brand name, is not a specification: it stays a plain field on the product's or the order line's snapshot

global reference (platform-owned, no tenant key):
  countries, currencies, ports, carriers, vessels

Resolution order: identifiers, then scoped aliases, then exact matches on standardized names, then finding possible matching records with a calibrated ranker over retained features. A model chooses only between closely matched candidates, and it gets the document's dates. Match, create and correct are three decision kinds; auto-creating master data (authoritative supplier, product, port, carrier, and vessel records) is a records are created only when configuration permits it; extraction alone cannot create them; extraction has no create path. Product resolution reads the alias table, which fixes the single most expensive dead end in the current system: aliases written on every review and read by nothing.

Two things today's organizations table does are pulled apart. The tenant, the account that logs in, keeps the table and its name, so organization_id keeps its meaning on every table and in every row-level-security predicate. Everyone a tenant deals with becomes an objects row of kind partner, one per (tenant, legal entity), roles carried in its snapshot: IMP's Al-Ain and Zaytoun's Al-Ain are two rows even when they are one company, and a partner that later becomes a tenant is linked rather than merged. A partner with several branches or ship-to addresses is a kind location row parented to it. Contacts belong to a partner object and carry as many channel addresses as the person has; the resolver from an address to (tenant, partner, contact) is what gives a conversation's identity and a forward its origin. A person's role at a partner, buyer, sign-off, accounts, escalation, is an object_links row of kind contact_of scoped to whichever object (the partner, or one location) the role applies to, with {contact_id, role, effective} in its payload; a contact can hold more than one role, and a branch-scoped role wins over the group's. This is the addressing layer, deliberately kept to two tables plus the partner and location kinds, and no CRM features; a tenant that runs a CRM makes it the authority for partner and contact facts through objects.authority, the same way an ERP is for orders.

3.4 Objects, links and facts

Every business object a tenant's ERP might own, an order, an obligation, a shipment, a partner, a product, a price list, is one row in objects: the same four tables regardless of kind, a JSON Schema file per kind instead of a table per kind, and generated columns for the handful of fields that everything else joins on or that money and evidence need a database constraint to guard. A relationship that carries meaning or quantity is a row in object_links. A fact about an object that is not in its snapshot, what a carrier reported, what a supplier promised, what a customer's reply confirmed, is a row in facts, reduced to a current value by a registered, versioned reducer. This is the general case the current system built once for shipment tracking; every business object gets it now, because messages are a first-class input and every object eventually acquires a second reporter.

objects                             -- one row per business object of any kind; the ERP's mirror, and for a tenant without an ERP, the record
  id PK, organization_id, kind, parent_id -> objects NULL,
  authority (aradus|erp), erp_integration_id NULL, erp_remote_id NULL, erp_watermark NULL,
  snapshot jsonb, schema_version,
  number text, counterparty_id uuid, currency text, total numeric, status text, effective_on date,
  -- generated from the snapshot, indexed; not every kind uses every column
  created_at, updated_at
  UNIQUE (organization_id, kind, COALESCE(counterparty_id,''), number) WHERE number IS NOT NULL
  -- counterparty_id in the key so two suppliers can each have an invoice numbered 1001: this is the
  -- obligation chain identity today's two inconsistent implementations collapse into, generalized to every kind
  UNIQUE (erp_integration_id, erp_remote_id) WHERE erp_remote_id IS NOT NULL
  UNIQUE (parent_id) WHERE kind = 'obligation_revision' AND status = 'current'   -- one live head
  -- kinds: order, order_line, obligation, obligation_revision, obligation_line, payment, adjustment, shipment, shipment_leg,
  -- handling_unit, freight_request, freight_request_line, freight_quote, freight_booking, commercial_request, quote,
  -- quote_revision, product, partner, location, lot, price_list, price_list_item, document
  -- each kind has a JSON Schema file in the repo, versioned by schema_version; a write that fails its schema is rejected

object_links                        -- every relationship that carries meaning or quantity; append-only, superseded not deleted
  id PK, organization_id, from_id -> objects, to_id -> objects,
  kind (allocation|reserves|finalizes|applies_payment|customs_copy_of|revises|pins_specification|evidence|source|located_at|contact_of|substitute_for|packaging),
  payload jsonb, decided_by_decision_id NULL, superseded_at NULL
  -- to_id is a FK to objects for every kind but two: contact_of and pins_specification reference a contacts or
  -- specification_versions row instead (neither is an objects row), carried in payload and checked in code instead of by FK

facts                               -- what anyone said about an object that the snapshot does not hold; append-only
  id PK, organization_id, subject_id -> objects, attribute_definition_id -> attribute_definitions,
  value jsonb, modality (contracted|expected|actual),
  effective_at NULL, observed_at, source_kind (document|provider|message|person|erp|model), source_ref,
  decision_id NULL, payload_hash
  UNIQUE (organization_id, payload_hash)   -- the same report arriving twice is one fact

attribute_definitions               -- immutable per version; a change is a new row
  id PK, organization_id NULL, subject_kind, name, version,
  value_type (text|integer|numeric|date|timestamp|money|code), reducer, reducer_version,
  actual_counterpart_id -> attribute_definitions NULL   -- e.g. order_line.ship_by pairs with shipment.shipped_on_board
  UNIQUE (COALESCE(organization_id,''), subject_kind, name, version)

current_facts                       -- materialized view: (subject_id, attribute_definition_id) -> current value, modality, reducer_version, derived_from

What the database enforces per kind: a unique business number within its kind, the ERP mirror's remote-id uniqueness, and, for the one case with money and a legal ordering requirement, the partial unique index that gives an obligation exactly one current revision. Everything else a kind needs, line items, addresses, charges, route legs, is inside the snapshot, validated by the schema file rather than by a column.

Commercial and freight. An order is an objects row of kind order, side carried in the snapshot, with order_line children; a request for quote and its quote are commercial_request and quote, a quote's own versions a child kind quote_revision the same way an obligation versions. Freight keeps its own lifecycle, comparison, ranking, equipment, routing, as its own kinds: freight_request with freight_request_line children frozen at send (the revision that went to the forwarders), freight_quote per forwarder reply, freight_booking once one is chosen. A quote comparison is a read model over sibling quotes. The seams to the rest of the model are object_links: an allocation ties an order line to the shipment or container carrying it and, once the first food or pharma tenant needs it, to a lot; reserves and finalizes tie a freight request's lines to the chosen quote and booking.

Logistics and tracking. A shipment is an objects row of kind shipment with shipment_leg and handling_unit children. Tracking is the first and best-known reducer, and its incident rules stay versioned code invariants with fixtures exactly as before: origin port write-once; destination changed only when empty or when the five reroute checks pass; provider telemetry outranks a document's dates; once an actual milestone is confirmed, later inputs cannot change it; BOL refreshes cannot overwrite operator edits. What changes is only the storage: every source's report is a facts row (subject the shipment or leg, attribute registered, modality expected or actual), and current_facts holds what the reducer computed, with derived_from pointing at the facts that produced it. A supplier's email pushing a lead time out a week is a fact on an order's ship_by attribute; a customer's reply confirming a quantity is a fact on an order line; a forwarded file is a fact that promotes a specification version. The contracted value, the date on the bill of lading or the order line's promised date, is written once from its document with an evidence link; the reducer does not touch it, and the operational value lives only in current_facts. An attribute_definition's actual_counterpart_id is what lets a promised date and the tracking milestone that actually settles it be compared without an analyst remembering which event means what. Message-sourced facts reach current_facts only through the review gate, so a misread email cannot move state without a policy or a person having seen it.

Settlement. An obligation is an objects row of kind obligation, and its obligation_revision children are the one place the design spends a database constraint on a business invariant: UNIQUE (parent_id) WHERE kind = 'obligation_revision' AND status = 'current' gives it exactly one live head, demoted in the same transaction a new one is promoted. obligation_line rows sit under the current revision. A document's own doc_class (proforma, commercial, customs, credit_note, debit_note) decides whether it can create an obligation at all: a proforma or a customs-only copy is evidence, linked by object_links kind evidence or customs_copy_of, and only the commercial invoice materializes one. A credit or debit note is an adjustment object bound to the obligation it corrects, with the posted revision it legally adjusts named in its snapshot. Payment is its own object, unbound from any one obligation, since a payment can cover part of one invoice and all of another: an applies_payment link, append-only, ties a payment to the obligation it settles, with the amount in its payload, so an unapplied balance is the payment's total minus its live applications and a wrong application is reversed by a new link rather than edited. The materiality gate and the ordering ladder that decide whether a new document is a new revision or a no-op are invariants with fixtures, unchanged in spirit from the current system's two inconsistent chain-key implementations, now one rule over one kind.

What is deliberately not built yet, because no committed tenant's flow writes it inside the next two migration steps: tenant use of the lot kind beyond its schema file and the allocation link's nullable lot_id; reservations for contended stock; sku_schemes, product_attribute_values and substitution_rules for segmented product codes; a per-tenant, per-entity, per-operation sync-direction contract beside objects.authority. Each is a design note with its shape sketched, so the addition is a migration and not a rethink.

3.5 Eventing, ERP, policy

Tying a document to what it created or supports is two of the object_links kinds from 3.4: source for provenance ("this BOL created this shipment", which does not project a field) and evidence for support, with its own is_primary flag in the payload where a single-FK projection needs one winner. A document is an objects row of kind document like anything else it might point at, so the link is objects-to-objects with no separate table.

domain_events      id PK, organization_id, event_type, subject_type, subject_id, payload, trace, created_at
  -- append-only log written by activities in the same transaction as the fact. A record for audit,
  -- analytics and debugging rather than a transport: side effects run as workflow activities, and a
  -- fact reaches a workflow by SignalWithStart after commit (see section 6, consistency without an outbox).

erp_commands       id PK, organization_id, integration_id, command, payload,
                   status, attempts                    -- durable command ledger
erp_links          integration_id, entity_type, entity_id, remote_id,
                   watermarks, sync_status (incl. parked conflict)
  -- superseded by objects.authority/erp_integration_id/erp_remote_id/erp_watermark for anything that is an objects row;
  -- kept for the handful of things ERP sync touches that are not, such as a raw inbound watermark ahead of its first pull.

policies                            -- the config surface; versioned, effective-dated
  id PK, organization_id NULL, counterparty_id NULL, counterparty_role NULL,
  document_type NULL, layout NULL, channel NULL, field_path NULL,
  origin_country NULL, destination_country NULL, origin_port NULL, destination_port NULL,   -- trade-lane scope: the axes compliance rules need
  product_id NULL, hs_prefix NULL, freight_mode NULL, direction NULL,
  policy_type (template|check|matching|autonomy|linkage|versioning|review_routing|model_routing|authority|sync_contract|step_gate),
  definition jsonb, version, effective daterange, created_by
  -- autonomy definitions carry a MODE as well as thresholds: auto_when_signals_agree | require_approval | ask_below_similarity
  -- model_routing: provider and model per task for this tenant (in-country transcription, a self-hosted language model); model_invocations records what was used
  -- sync_contract: per kind, which operations each side may perform (create, update, which fields), in which direction, and what happens on conflict.
  --   authority says who owns the fact; sync_contract says what may cross the boundary. Quotes and asks are created here and pushed; price lists are pulled;
  --   an ERP tenant's order may be created here from a forwarded PO, pushed, and corrected there afterwards.
  -- step_gate: which optional steps of a named workflow variant run for this tenant or counterparty
  -- authority: the per-tenant default for objects.authority by kind, e.g. {kind: order, authority: erp}; a new objects
  -- row without an explicit authority reads this policy at write time, defaulting to aradus when none is set
  -- every policy_type declares its runtime reader; a test fails if a surfaced
  -- option has no execution path. Incident rules are NOT here: they are
  -- versioned code invariants; policy may route outcomes to review, not weaken them.

audit_events       subject_id -> objects, actor, action, before_ref, after_ref

Who owns a fact is now a column on the row itself, objects.authority, rather than a side table keyed by entity type: for a tenant like RVK every commercial kind reads erp from its authority policy row at creation and Aradus does not originate one; for a tenant with no ERP the default policy says aradus and being the record of their orders means being their ERP. Deliveries, dispatches, artifacts, proposals, decisions, evidence, policies, specification versions and asks are Aradus-owned for every tenant and carry no authority policy at all.

3.6 What moves, what stays

One rule decides the fate of today's 150 tables: a table moves to the new schema only if the design changes what it means. Everything else stays in public and is reached through it.

Twenty-five tables carry the design, plus the tenancy, auth and reference tables that stay in public unchanged.

4 · Workflows

Every capability is a Temporal workflow written in code, and there are three shapes. Entity workflows live as long as the thing they are about: one per conversation, one per shipment, one per ERP integration. Run workflows do one unit of work and end: a file through the pipeline, a document through extraction, an order through confirmation, a loading list through generation. Schedules replace cron: digests, sweeps, retention, billing close. Model calls are activities and stay out of workflow code. Every value a tenant varies is read from policies at the step that needs it. Nothing an operator can draw; everything an operator can configure.

flowchart TB
    D[deliveries
email, WhatsApp, upload, API] --> CW[ConversationWorkflow
entity: tenant, channel, contact]
    CW -- a question --> CW
    CW -- answers an open question --> ASK[the workflow that asked]
    CW -- a file or attachment --> FR[FileRun
classify and split]
    FR --> DR[DocumentRun
extract, resolve, link, version]
    CW -- text about an object --> DR
    CW -- a forwarded design file --> AR[ArtworkReplacementWorkflow
run per file]
    SCH[schedules
forecast refresh after the ERP pull] --> FC[ForecastRun
demand, lead time]
    Q[ERP screens and Ask AI
a person typing] --> SG[SuggestionQuery
synchronous, from cache]
    DR --> GATE{review gate
policy or a person}
    AR --> GATE
    FC --> GATE
    SG --> GATE
    GATE --> MAT[materialize
objects, facts]
    MAT --> OC[ConfirmationWorkflow
run per order, per ask kind and policy]
    MAT --> TR[ShipmentWorkflow
entity per shipment]
    MAT --> ERP[ErpSyncWorkflow
entity per integration]
    P[provider feeds
tracking polls, ERP pulls] --> TR
    P --> ERP
    OP[operator actions
start, decide, generate] --> OC
    OP --> DG[DocGenRun
loading list to documents]
    OC --> NT[dispatches
notifications, questions, documents out]
    TR --> NT
    ERP --> NT
    DG --> NT
    DS[DigestSchedule and housekeeping
digests, sweeps, retention, billing] --> NT
    NT --> OUT[to people and systems
email, WhatsApp, ERP]

Arrows are "wakes" or "writes". Every workflow writes through activities to the same tables; the gate is the same code path for a field, a link, a version, a sign-off or an assertion.

Synchronous surfaces. One family is not a workflow. A product-code suggestion under a cursor in the ERP, or a question typed into Odoo's Ask AI, must answer in well under a second, and a Temporal round trip is the wrong tool for that. SuggestionQuery is a plain request handler over a per-tenant cache: a frequency table of (customer, user, code prefix) refreshed by a schedule, and the alias and identifier tables. It still writes what every other family writes: a runs row, an entity_match proposal per suggestion, and a proposal_decisions row when the person accepts or types something else. That is what makes the acceptance rate measurable and the suggestion learn per customer. A model call is allowed on the refresh and not on the request.

5 · Key flows

The trace in full: one invoice, every write, in order

ACME (partner ptn_9a2) emails invoice INV-4471 for USD 4,200 against shipment shp_113. Every row the pipeline writes, numbered in commit order, showing the values stored in each row. Each row contains IDs linking it to the preceding rows so each write is traceable to the ones before it.

Reads are omitted; the diagram shows writes only, in commit order. Proposals (6, 11–13) and decisions (7, 14, 18) are the only tables written in more than two steps, and both only ever gain rows.

Merged PDF, five documents inside

1. Intake creates one delivery, one artifact, and a file-level run, opened before its documents are split (the CHECK on runs allows exactly this case). No document row yet: nothing is known to be a document.
2. Classification proposes each document's type and page range, one proposal per segment with its confidence. Policy decides them; a low-confidence segmentation pauses for review before anything is sliced.
3. Materialize: per decided segment, create a derived file for each approved page range into artifacts as a rendition (parent_artifact_id set, kind page_png), insert a documents row with artifact_part and the parent delivery's provenance, and start a per-document run. A redelivered split re-runs against the same digests and no-ops.
4. Each document proceeds through the happy path independently. A reclassification of the artifact replaces the earlier set while retaining it in history.

Workbooks follow the same flow with sheet names as artifact_part. There is no parent pseudo-document and no fake system actor: The delivery record carries the source information.

Denied segmentation: retry and correction

What happens when the split proposal is wrong. The cost depends entirely on timing: before documents exist, denial is free; after, it is a generation supersession. The gate exists to make the second case rare.

flowchart TD
    P["segmentation proposal<br/>on the file-level run (pass 1)"] --> D{"decision"}
    D -- "reject: wrong, machine<br/>should try again" --> E["escalated re-read:<br/>higher DPI, stronger model"]
    E --> P2["new proposal, pass+1,<br/>same run - prior pass kept"]
    P2 --> D
    D -- "defer" --> W["file run stays<br/>waiting"]
    W --> D
    D -- "auto-accept<br/>(policy threshold met)" --> M["materialize: documents born typed<br/>+ per-document runs"]
    D -- "correct: the reviewer's boundaries<br/>become the decided_value" --> M
    M --> OK(["pipeline continues<br/>per document"])
    OK -.-> L["split found wrong LATER,<br/>documents already exist"]
    L --> R["explicit resegment:<br/>new file-level run"]
    R --> NP["new segmentation proposal"]
    NP --> ND{"decision"}
    ND -- "accept / correct" --> G["generation+1 documents,<br/>born typed from the corrected split"]
    G --> SUP["old generation superseded, not deleted:<br/>runs, proposals, decisions, links kept;<br/>materialized records reconciled by supersession"]

Reject cannot loop unbounded: each escalated pass produces a new proposal that costs a fresh decision, and correct is the terminal fallback - the reviewer states the truth and documents are created from it. Every wrong guess stays on record as a labeled example.

Duplicate arrival, and re-upload with intent

If the resent file has the same hash, reuse the existing file record: a new deliveries row records the arrival (sender, channel, time) and nothing reprocesses - existing documents keep their original producing delivery and are not retargeted. A manual re-upload whose request metadata states what the operator wants also reuses the stored file: a reprocess request opens new runs on the existing documents; only an explicit reclassify or resegment creates a replacement document generation. Content is stored once in both cases; the difference between "we received it again" and "the operator wants it re-read" is a field on the delivery row rather than a code path.

Revised bill

The new document resolves to the same obligation by its chain key, (organization_id, kind, counterparty_id, number). The materiality gate compares with the current revision: an empty or trivial diff returns the head unchanged, so a re-upload cannot reset an approved bill. A material diff creates a revision proposal; the decision (policy or human, per the tenant's versioning policy) inserts a new obligation_revision object as a child of the same obligation, and the partial unique index (parent_id where status = 'current') demotes the old head in the same transaction that promotes the new one. Payments are their own objects linked by applies_payment to the obligation itself rather than to one of its revisions, so a revision change cannot re-bind them: the outstanding balance is the current revision's total less every live application, and an old revision is superseded without disturbing the money already applied.

BOL creates a shipment

1. Extraction and resolution produce proposals for the BL number, parties, ports, containers, and an entity_create proposal for the shipment itself, since no matching shipment exists.
2. On acceptance, commit create,s the shipment, its legs and handling units (trackable containers, packages, or pallets) in one transaction, with the materializations digest guarding redelivery. A container number that fails its check digit materializes the unit with a null identifier and a pending field task; keep the suspected number only as an unapproved proposal; do not use it for tracking.
3. An object_links row of kind source records that this document created this shipment. It does not populate any derived shipment field; a duplicate BOL arriving later gets an evidence link only.
4. Tracking initialization appends the first facts; the reducer computes initial state; from here provider polling and the BOL disagreeing about dates is the reducer's problem, under its versioned precedence rules.

Ambiguous shipment link

Two candidate shipments for one certificate: The proposed match stores both candidate shipments and the evidence used to compare them including the document's own dates, so a container reused eight months apart resolves without a person. A genuine tie leaves the link proposal pending; the human review task shows both candidates with evidence; the decision creates the evidence link, and derive updates the projections. A later manual re-link supersedes that specific link and re-derives. No scheduled retry calls and charges the model again while review is pending while the task sits open.

A correction that teaches

1. An operator corrects the seller on a pending entity-match proposal: the decision row records decided_value beside the untouched proposal, the actor, and the task.
2. The same transaction writes a scoped alias ("this counterparty's 'ACME TRDG LLC.' means party X") with created_from_decision_id, subject to the an alias may identify only one record; a conflicting alias opens a review task; a collision with an existing alias opens an ambiguity task instead of silently overwriting.
3. The proposal-decision pair becomes a labeled example: it joins the calibration set for that field and counterparty, and the few-shot pool for the next document from that sender (see "Three ways a correction is used" in section 6), with evaluation cases kept separate.
4. The next invoice from this supplier matches the seller exactly from the saved alias before calling a model, before any model runs.

Reprocess and change type

Reprocess opens a new run; the old run and its proposals stay, marked superseded. New proposals are decided (prior human decisions auto-apply where subject and value match; the rest re-enter review), commit re-materializes through the same digests, updating rows in place where the target exists, and evidence links are superseded and recreated rather than deleted. A change of type does not update document_type in place: it is a human-decided classification proposal that creates a replacement document generation, supersedes the old one, and reconciles its evidence links and materialized records. Nothing is hard-deleted, so a reprocess cannot destroy corrections, and the a safeguard requiring approval when changing the type would leave a shipment without its creating document is a policy rule: a type change that would orphan a shipment routes to an approval task instead of being forbidden in code.

ERP two-sided conflict

1. The inbound pull detects that both Aradus and the ERP changed the same record since the last sync. The erp_links row is marked as a conflict; the conflict remains paused until a person resolves it; routine synchronization must not resume it, since the park's own bookkeeping must not un-park it on the next sweep.
2. An ask of kind erp_conflict, audience user, opens with both versions and the field diff. No writes flow in either direction while it is open.
3. The operator picks theirs, ours, or a merge. The decision releases the conflict, and the outcome becomes an erp_commands row (ours) or an inbound patch (theirs), each idempotent under the customer's compatibility contract.

Text-only message about an existing object

1. A carrier emails "MSC Adriana will discharge at Khalifa Port, ETA 27 March." The delivery has one body artifact; classification proposes correspondence and the document is born typed.
2. The run produces a refers_to proposal and two assertion proposals: destination port = Khalifa (actual) and ETA = 27 March (expected), each with a text-span source_ref.
3. Policy for this tenant trusts carrier senders about routing, so the port assertion auto-accepts and materializes a fact. The ETA assertion auto-accepts as expected, and a notification fires because the slip exceeds the tenant's two-day rule. Both materializations key on the shipment instance, so a second email about a second container updates a second shipment without collision.
4. When the provider feed later reports the actual berth, that fact outranks the expected one and current_facts moves. The email's fact stays on record as what the carrier said.

A person messages first

A contact sends "cancel line 3 of yesterday's order" with nothing pending. Intake does SignalWithStart on conversation-{tenant}-{channel}-{contact}. The conversation workflow's interpret activity classifies the message as an instruction about an object, resolves "yesterday's order" against the contact's recent deliveries, and produces an assertion proposal (line 3 status = cancelled) that goes through the same gate as any other. Had the message answered an open question, the conversation would have signalled the workflow that asked. Had it been a question, an activity would have read current state and drafted a reply as a dispatch. Every turn is a delivery or a dispatch row; the workflow holds only the open questions and pointers to the last few turns, and continues as new after a bounded number of turns.

6 · Deep dives

One way to ask a person

Every question put to someone outside the machine needs the same four things on record, whatever the question is about: what was asked, item by item; the answer, in the words that gave it; what happens while no answer has arrived; and what stays blocked until one does. A customer confirming artwork on an order, a supplier acknowledging a purchase order, a new supplier sending onboarding documents, and an operator settling a field the model was unsure of are the same shape underneath: one asks row for the set of questions, one ask_items row per thing asked about, and a policies row that says how often to chase, when to escalate, and what counts as an answer. The three kinds of decider, a person on staff, a counterparty replying from outside, and a policy that settles the question without asking anyone, are the three actor_type values already on proposal_decisions; the policy kind needs no asks row at all, because nothing was ever pending.

Camlica acknowledging a purchase order for jars is one asks row of kind po_acknowledgement, addressed to Camlica's sales contact, with one ask_items row per PO line carrying quantity, price and ship-by date. Camlica's reply confirms the jars and lids lines and moves the freight line's ship-by date out two weeks; that reply becomes a proposal_decisions row of confirmed on that item, with the counter-date carried in the decided value and the actor set to counterparty. The moved date is also a fact about the order line in its own right, so it becomes a fact the same way a supplier's unprompted email would, and that is how the slip reaches the customer's promised date and the freight booking without either one reading a single row of the ask.

Confidence and calibration

The gate does not read a model's self-assessment. Inputs to the calibrated score: cross-pass agreement (pass 2 uses a different model or rendering, since two correlated passes are weak evidence), deterministic validator results, resolver features, and the counterparty's track record. The calibrator maps raw scores to observed correction rates (adjusts model scores using how often similarly scored values were actually corrected) per document type, field and layout. Its training data has three sources by design: gold cases (manually verified examples with known correct answers), production corrections, and random audits of auto-accepted work, because a dataset containing only corrections is biased toward cases that were already sent to reviewers. Hierarchical priors cover cold starts; retrieval examples are kept out of evaluation sets.

How the calibrator is trained. It is not a language model and nothing is fine-tuned. The training set is every decided proposal in scope: the features recorded on the proposal at the time (raw model score, whether the two passes agreed, validator results, resolver features, the counterparty's history) and one label, whether the person left the value alone. A correction or rejection is a zero; an accept is a one. Three kinds of decision are allowed in: the ones routed to review because they scored low, a random sample of auto-accepted work that a person is asked to check anyway, and gold cases with known answers. The random sample is not optional. Without it the set holds only low-scoring cases and the mapping learns nothing about the scores it is meant to trust. Decisions flagged eval_holdout are excluded and used only to score the result.

Fitting is a monotone mapping from raw score to observed correctness. At today's volume, tens of documents a day across all tenants and a few hundred labeled decisions per field per month at best, the fit is one small logistic model per document type and field at the platform level, and nothing finer. Isotonic regression and per-layout or per-counterparty scopes need more points than that to be anything but a noisy step function, so they wait for the data. A tenant gets its own mapping for a field only once it has crossed a fixed count of labeled decisions there, on the order of 150; below that it reads the platform mapping. Nesting with shrinkage between the levels is the later refinement, once enough tenants have enough data for pooling to beat a hard cutoff. That, in stages, is what "hierarchical priors cover cold starts" means. Fitting runs in Python with scikit-learn's calibration tools; what ships to the Go workers is the fitted table or coefficients, which apply as a lookup. The output is a fitted_models row: the mapping, what it was trained on, and a backtest on the held-out decisions with two numbers, calibration error, and the error rate of what the mapping would have auto-accepted at the threshold. A new version replaces the old only if the second number does not get worse, and only when the training set has grown by a minimum count since the last fit, the same promotion discipline ABBYY documents for its online learning (a minimum of thirty documents, a held-out split, promotion on measured gain). Retraining runs on a schedule and every proposal records the version that scored it in produced_by, so a change in accuracy can be traced to a calibrator change as well as to a model or prompt change. The threshold stays where policy put it; what moves is the score.

Three ways a correction is used

One correction feeds three mechanisms, and they do not overlap. An alias replaces the model for a string it has seen: written in the decision's transaction, read by the resolver before any model call, deterministic. Few-shot examples make the model better on a layout it has seen. Calibration decides when the model may act alone, on evidence, whatever the model is. The first two improve the answer; only the third decides whether to trust it.

Few-shot examples are a query over existing rows. An example is a decided proposal joined to its evidence: the page crop or text span from a rendition artifacts row and source_ref, the field, what the model proposed, what the person decided. Before extracting a document the gateway selects examples for each field in this order: same tenant and same sender first, then same document type and layout, most recent first, corrections preferred over plain accepts because they carry the most information, capped at k per field, with eval_holdout rows excluded. The selected decision ids are written to the proposal's produced_by.few_shot_examples, so an extraction can be replayed with the examples it saw and a change in accuracy can be attributed to a change in examples as well as to a model, prompt or calibrator change. The pool needs one index, on (organization_id, counterparty, document_type, field_path, decided_at), and nothing else.

What the market does. The routing pattern here is the one the document-processing vendors use: a per-field confidence checked against a threshold grounded in observed accuracy rather than the model's own estimate (Rossum, Hyperscience), evidence spans on every value (all of them), and a random sample of accepted work sent to a person anyway (AWS Augmented AI documents it as a use case). Where this design goes further is in keeping the proposal and the decision as separate immutable rows, where every vendor overwrites the value, and in learning through aliases and examples rather than retraining a model. No surveyed tool, Langfuse and LangSmith included, fits a calibrator, selects few-shot examples by sender and layout, or acts as a record of decisions that drive business writes. The review surface is not new work either: the document page and the Command Center cards already exist and are customer-facing; the migration re-points them at asks and proposal_decisions. Langfuse stays what it is, tracing for engineers, and is not a place where decisions are made.

Policy versus invariant

Configuration is applied from platform defaults down to field-specific rules (platform → tenant → counterparty and role → document type → layout → channel → field). It holds statements of business fact: templates with field-to-column maps, checks with tolerances and severities, matching strategy order, linkage conventions, materiality rules, autonomy grants. Use centrally managed thresholds based on observed error rates; customer-specific thresholds would overfit small datasets. Incident-derived correctness rules are not policy at all: the ordering ladder, tracking precedence and ERP idempotency are versioned code with fixtures, and policy can only choose whether an outcome routes to review.

Idempotency where money moves

Three ledgers make repeated delivery harmless (three durable operation records prevent retries from repeating billable or financial effects). model_invocations: a call is reserved under a deterministic ID before dispatch, so a crash after the provider answered is reconciled, not re-billed, and billing reads this table rather than queue traffic. materializations: A repeated commit matches the existing input hash and makes no change. erp_commands: every push is a durable command with attempts, and the customer-facing semantics (external ID schemes, for products first created in the ERP, treat an exact SKU match as the same product, case-insensitive number matching within tenant and direction, draft-only posting, already-posted no-ops, line replacement, conflicts that stay paused until a person resolves them) are a per-customer compatibility contract, changed only after certification against that customer's system.

Messages as input: six changes

Documents arrive with their own identifiers and a fixed shape; messages do not. Six changes make a message a first-class input rather than an attachment carrier. Bodies are artifacts and deliveries carry threading (3.1). Linkage is its own proposal type, refers_to, with its own threshold, because for a message the hard question is which object it is about rather than what it says (3.2). Evidence anchors include a text span, so a review shows the quoted words rather than a page (3.2). Statements about existing objects are assertion proposals that materialize facts, and the reducer covers every subject type rather than only shipments (3.4). Materialization is keyed by target instance, so one message can update several objects of one type (3.2). Extraction schemas for correspondence are assertion-shaped: a variable-length list of (subject, attribute, value, modality, effective time) rather than fixed fields, declared as a document type whose schema allows a list. The delivery is the unit of context for all six: one message, one body, N attachments, read together.

Who owns which fact

Two kinds of record are easy to confuse. The ERP records business facts: an order, an invoice, a customer. Aradus records how a fact came to be: the message it arrived in, what the machine proposed, what a person decided, the evidence, and the rule that came out of it. The second kind is ours for every tenant and is the memory that makes the system better per customer; no ERP holds it. The first kind is owned per tenant and per kind through objects.authority. When an ERP is connected it originates and corrects the fact, our row mirrors it with the remote id, and an accepted proposal becomes an erp_commands push rather than a local write. When no ERP is connected, we are the record, edits happen here, and an ERP added later is a downstream target. Same tables in both modes; the flag decides who may originate a fact and which way sync runs. The consequence is accepted: for a tenant with no ERP, being the record of their orders means being their ERP.

Conversations are workflows

Headless and conversational are the same step at two points on the confidence scale. A clean spreadsheet order and a photo of a handwritten one both produce a sales-order proposal; one clears the tenant's threshold and materializes, the other becomes a question. The question is a dispatch to the person on their channel, the reply is a delivery, and the reply is read as a decision on the open proposal. So a chat is a workflow: state is the set of open questions and pointers to recent turns, an inbound message is a signal, a model call is an activity (one to phrase the question, one to read the answer), an outbound message is a dispatch. There is no separate agent; the workflow is the agent, and headless is the same code with the question step skipped. Messages a person starts land in a per-contact entity workflow keyed conversation-{tenant}-{channel}-{contact}, created by SignalWithStart, which routes each message as an answer to an open question, an instruction about an object, or a question to be answered from state. Its id gives the tenant discrimination the current WhatsApp routing lacks. Automated feeds stay on the facts path and are kept out of conversations, because a workflow processes its signals one at a time.

Runtime: self-hosted Temporal, control flow in code

Temporal is self-hosted on the existing Kubernetes cluster for data residency, with the CNPG Postgres operator as persistence. It is a durable execution runtime for code and offers no visual authoring layer. The current system's Engine B bundles two things: durable execution, done by hand with rows, a poller and a seven-day stale sweep, and a graph builder operators can edit. The greenfield keeps neither. Control flow is code, as Temporal workflows, versioned like code: match, send, wait, remind, convert is the same shape for every tenant. Everything a tenant varies is policy data read by the workflow at each step: who to send to, how often to remind, which threshold auto-accepts, whether a marked-up screenshot counts, which mode a replacement runs in (auto_when_signals_agree, require_approval, ask_below_similarity). The operator-facing surface is a rulebook rather than a canvas. Model calls live only in activities; workflows carry ids rather than customer data; long chases continue as new; and worker versioning is decided before the first workflow runs for weeks, since a deploy that changes a running workflow's logic without it fails on replay.

When the ERP owns everything

For a tenant like RVK, the authority policy sets objects.authority to erp for every commercial kind: quotation, order, purchase order, receipt, product. Aradus then does not originate a commercial document. It reads attachments off the ERP record through the inbound pull, proposes, lets a person decide in our review surface or in the ERP's own activity, and writes back through erp_commands: a staging line, a flag, a mail.activity for the right team, or a value already approved. Three things follow. First, what we own is exactly what the ERP has no model for: decisions with their evidence, drawing and audio artifacts, fitted models with their backtests, supplier promised-versus-actual history, the acceptance record of every suggestion. Remove Aradus and the ERP still trades; remove it after a dispute and the evidence is gone, which is the honest description of the moat. Second, the ERP's API has no multi-call transaction, so an erp_commands row is one call, idempotent through the ERP's external id, and a pair that must land together (write the extracted lines, then create the activity) is a small saga in the workflow with the second step retried until it lands and a compensating command if it cannot. Third, history the ERP may not carry over from a migration decides whether a forecast ships warm or cold, so the pull that feeds fitted_models records what it found and the design makes cold start a state rather than a failure.

Consistency without an outbox

A Postgres commit and a Temporal call cannot be one atomic operation, and the design uses Temporal's own answer rather than a relay. For actions a person takes (Start, a decision, remind now) the API updates or signals the workflow and the workflow's activity writes the row: the workflow is the source of truth while it is open, and Postgres holds the projection the UI reads. Temporal's founder gives this as the recommended pattern. For arrivals from outside the handler writes the delivery idempotently on the provider's message id, commits, then does SignalWithStart on the deterministic workflow id with reuse policy reject-duplicate. The provider retries until it receives a 2xx, so a crash between commit and signal is healed by the retry hitting the same idempotent write and the same idempotent start. Temporal's guidance for this case is the same: an idempotent transaction plus an idempotent start needs no outbox. What is given up: a mutation depends on Temporal being reachable, any writer that bypasses the application must start workflows explicitly, and deduplication moves from one relay into every activity. What is gained: no relay, no broker on this path, and replay by resetting a workflow. domain_events stays as an append-only record written by activities, for audit and analytics, and is not read by anything to trigger work.

7 · Migration plan

The company is under six months old and the current system has two paying tenants, so this is a plan for moving a small live system in pieces rather than cutting over a large one. Decisions that shape it: the core moves and the edges stay; capabilities move in dependency order and are switched on per tenant; the graph builder is frozen from today; history moves forward from each cutover; the greenfield gets a new schema on the existing Postgres instance.

End state

Everything that reads, decides, remembers or waits runs on the spine: intake as deliveries, the document pipeline, invoice versioning, tracking reduction, every workflow as Temporal code, memory as proposals, decisions, facts and policies. Everything that is an adapter to the outside is kept and re-pointed: the Next app with its auth and RBAC, the provider clients (SeaRates and the other tracking vendors, Twilio, Resend, Graph), the ERP connectors, the doc-gen renderer, Slack. Old Node and Go business logic is deleted when its last reader has moved; adapters move into the new codebase afterwards as file moves. Engine B is frozen now: no new graphs or node types, existing graphs run until each is rewritten as a workflow.

Order

Forecasting and suggestions are not migration steps. They are new families on the spine and can start at any point after step 3, gated per tenant like everything else.

Steps 1 and 2 are the seams every later step shares, which is why they go first and why they move behaviour without improving it: every tenant's traffic passes through them. From step 3 on, each step is a set of readers moving to tables that already hold production data.

History

Each capability starts on the spine with current state: live documents, current specification versions, open orders, and one baseline fact per shipment labelled as a legacy baseline. Older rows stay in public, readable through a legacy view in the Command Center and reports, and are left where they are. Provenance stays honest because nothing is invented, and the old tables are dropped when retention ends rather than after a port.

Database

One Postgres instance. The greenfield owns a new schema with its own migrator and row-level security on every table; nothing in it references public by foreign key. The Next app and the Go API reach both schemas over one connection, so a read model that shows an old order beside its new sign-offs is one query during the transition. The two-migrator drift that produced the ARD-330 incident cannot recur, because each schema has exactly one migrator. Section 3.6 lists every current table and where it ends up.

What parity means

Parity is user-observable capability, listed before anything is built: channels, duplicate rules, file types, document types, fields, resolutions, mutations, links, reviews, statuses, notifications, metering, tracking rules, ERP operations, retry and give-up paths, permissions, audit. Incident rules become characterization tests carrying their original examples. Table shapes and the current system's two inconsistent ways of versioning a bill are implementation detail, not parity. The eval corpus and CI gates exist from step 3 onward; each capability adds its gold cases before its model calls ship.

Produced 4 Sep 2026 against branch next; revised 9 Sep 2026 with messages as input, authority per tenant and entity, conversations as workflows, the self-hosted Temporal runtime, consistency without an outbox, and the migration plan in section 7. Companion pages: Order Confirmation Design and Aradus Field Guide. Earlier companions: Document Pipeline ERD and Pipeline Write Paths.