Greenfield Pipeline Design

1 · Problem and requirements

Aradus converts untrusted, counterparty-authored documents into shipment, payment, compliance, tracking and accounting records, and those records drive the customer's operations and their accounting systems. A perfectly extracted field can still be wrong in every way that matters: attached to the wrong shipment, matched to the wrong legal entity, duplicated into a second bill, treated as older than an outdated version, posted twice to an ERP (enterprise resource planning) system. So the product is correctly matching records, preserving their sources, resolving conflicts, selecting the current version, and writing only approved data. Extraction is the easy half.

Four kinds of record must stay separate. A delivery is how something arrived and from whom. An artifact is the file itself - the PDF or spreadsheet as received, stored once and not modified afterwards - possibly shared by many deliveries (the same attachment sent twice is one artifact, two deliveries). A document is a distinct business document within the file, created once classification has decided it exists; one artifact may hold five documents, and one document may arrive repeatedly or revised. A run is one interpretation using recorded, fixed versions of the code, model, prompt, schema, and policy (configured rules deciding what may be automated and what requires review). Most of the current system's complexity traces to collapsing these four into one table.

Functional requirements

1. Ingest via upload, email, chat, API (receive files through upload, email, chat, or API); merged PDFs split into documents; process each workbook sheet separately; store one copy of byte-for-byte identical files while recording every arrival with record the sender, channel, and time for every arrival.
2. Classify, extract, normalize (convert extracted values into standard formats) and validate roughly 25 document types; about nine of them create or update business records (invoice, bill of lading (BOL), air waybill (AWB), order, freight quote, packing list, two certificates, courier receipt); the rest are evidence.
3. Match extracted names and codes to supplier, product, port, carrier, and vessel records with alternate names that apply only to a particular customer or supplier; match, create and correct are three separate decisions.
4. Link documents to shipments and orders; rank candidates using identifiers, sender, dates, parties and route; send genuinely ambiguous matches for human review.
5. Version supplier bills: Check whether the changes are significant enough to create a new bill version, fixed tie-break rules for deciding which revision is current, payments remain attached to the bill's current version; revisions for every other document type too.
6. Route uncertain values to human review; every correction becomes training and evaluation data; approvals and corrections both become evaluation and training examples.
7. Synchronize records with Odoo and Business Central. (Odoo, Business Central) and tracking without changing the accounting behavior customers already rely on.
8. Reprocess and change type without destroying history; per-tenant (per customer organization) document type catalog; per-counterparty templates and conventions set up through configuration rather than code.
9. Record the usage and cost of every model call for billing, including failed calls.

Non-functional requirements

1. Multi-tenant with hard isolation (one shared system serving multiple customer organizations while strictly separating their data); a bug in one tenant's query cannot read another's rows.
2. At-least-once delivery (each message is delivered one or more times, so duplicates are possible) everywhere; every effect idempotent (safe to retry: repeating the same operation does not create a second effect); a billable model call is not repeated on redelivery.
3. Every accepted value traceable to its evidence, machine proposal, policy and decision, with the actor recorded.
4. Accuracy measurable per stage, per tenant, per document type; changes in accuracy can be traced to a model, prompt, or dataset change.
5. Automatic approval can be enabled or disabled by field, counterparty, and document type; gated on calibrated (adjusted so a stated confidence matches the observed error rate) confidence, not model self-assessment.
6. Evidence retained per tenant policy with legal hold (a requirement to preserve evidence because of litigation, investigation, or regulation); a customs dispute runs years.
7. Onboarding a new counterparty format costs configuration, not engineering.

Scale honesty

This is not a throughput problem (a limit on how many documents the system can process per unit of time). Current production creates on the order of one shipment a day; even at 100x, Postgres on one primary handles every table here without partitioning. The scarce resources are model spend, human review minutes, and correctness under retries. The design optimizes for those three, and for the growing value of the corrections dataset, not for request throughput.

2 · Architecture

Five components, and the reason each earns its place:

• One modular Go codebase deployed as API and worker pools (groups of background processes that execute queued work). No microservices: the transactional boundaries (groups of changes that must either all commit or all fail) around documents, obligation (a payable or receivable amount owed)s, shipments and reviews matter more than deployment independence at this scale.
• Self-hosted Temporal on the existing Kubernetes cluster handles execution: leases (time-limited ownership of a task by one worker), retries, timers, heartbeats, fan-out and fan-in, review waits, conversation signals, cancellation, and continuity across deployments. Postgres stays the only business truth; the UI and API read run status from Postgres projections (query-friendly views derived from the authoritative records), not from Temporal. Self-hosting is a data-residency decision and carries an operations cost: four services plus persistence, and someone on call who has run it.
• Control flow is code and tenant variation is policy. Workflows are written and versioned like code; every value a tenant changes (recipients, reminder intervals, thresholds, modes) is a policy row read at each step. There is no operator-editable graph builder; the operator surface is a rulebook. Conversations with people are workflows too, one per contact and channel, so headless and chat-driven steps share one code path (section 5).
• A Python media and model gateway, deliberately shrunk: PDF and image rasterization (PyMuPDF has no Go equal), provider adapters and their quirks, and the the process for testing and improving extraction prompts and examples (few-shot assembly (selecting a small set of worked examples to include in a model prompt), future future automated prompt optimizers). It is stateless (the service keeps no durable business data between requests), has no database credentials, and schemas are defined once in JSON Schema and used to generate matching Go and Python types, so the the Go and Python schema definitions becoming inconsistent the current system suffers cannot recur. If the PDF tooling gap ever closes, this component can move into the Go application; nothing else depends on its language.
• PostgreSQL, one primary: Authoritative stored data, row-level security provides a second enforcement layer between customers' data, transactional outbox (a database table of events written in the same transaction as the business change, then delivered reliably) with one delivery row per subscriber.
• S3: the original files (written once, then read-only) plus derived files such as page images and crops, which are regenerable copies rather than authoritative source data.

3 · Data model

erDiagram
    conversations ||--o{ deliveries : "inbound - one thread per tenant, channel, contact"
    conversations ||--o{ dispatches : "outbound - the mirror of deliveries"
    deliveries ||--o{ delivery_artifacts : "body and attachments alike"
    delivery_artifacts }o--|| artifacts : "many arrivals, one stored file"
    artifacts ||--o{ renditions : "derived page images and crops"
    artifacts ||--o{ documents : "artifact_id"
    deliveries ||--o{ documents : "delivery_id - the producing arrival, same file enforced"
    pipeline_proposals |o--o{ documents : "created_from_proposal_id UNIQUE - born typed"
    document_families ||--o{ document_family_members : "family_id"
    documents ||--o| document_family_members : "document_id UNIQUE"
    artifacts |o--o{ pipeline_runs : "file-level runs (classification)"
    documents |o--o{ pipeline_runs : "per-document runs - CHECK exactly one of"
    pipeline_runs ||--o{ pipeline_proposals : "run_id"
    pipeline_runs ||--o{ model_invocations : "reserved before dispatch - billing truth"
    pipeline_proposals ||--o{ proposal_decisions : "append-only verdicts - latest wins"
    proposal_decisions |o--o{ adjudication_tasks : "task_id when a person decided"
    proposal_decisions |o--o{ entity_aliases : "a correction writes a scoped alias"
    proposal_decisions |o--o{ evidence_bindings : "an accepted link creates the edge"
    documents ||--o{ evidence_bindings : "document_id"
    evidence_bindings }o--|| business_objects : "typed target - an enforced FK"
    pipeline_runs ||--o{ materializations : "accepted set digest - idempotent persist"
    materializations }o--|| business_objects : "the domain record it created"
    business_objects ||..o{ domain_aggregates : "orders freight shipments obligations..."
    organizations ||--o{ pipeline_policies : "the config cascade"

    deliveries {
        varchar id PK
        varchar organization_id FK
        intake_channel channel
        varchar contact_id FK "sender identity"
        varchar conversation_id FK "thread"
        varchar idempotency_key UK
    }
    artifacts {
        varchar id PK
        varchar organization_id FK
        text content_hash "UNIQUE per org"
        text storage_tier "cost"
        timestamptz retain_until "obligation"
    }
    documents {
        varchar id PK
        varchar artifact_id FK "NOT NULL"
        varchar delivery_id FK
        varchar created_from_proposal_id FK "UNIQUE - retry-safe identity"
        integer generation "resegment supersedes the set"
        text artifact_part "pages or sheet"
        varchar document_type "NOT NULL - born typed"
    }
    pipeline_runs {
        varchar id PK
        varchar document_id FK "nullable"
        varchar artifact_id FK "nullable - CHECK one of"
        run_status status "running waiting_review completed failed superseded"
        varchar policy_snapshot_id
    }
    pipeline_proposals {
        varchar id PK
        varchar run_id FK
        proposal_type proposal_type "classification field assertion refers_to entity_match link revision"
        varchar subject
        smallint pass "UNIQUE run+type+subject+pass"
        jsonb proposed
        jsonb source_ref "page bbox - what review highlights"
        numeric confidence "calibrated"
        jsonb produced_by "model prompt schema policy versions"
    }
    proposal_decisions {
        varchar id PK
        varchar proposal_id FK
        decision decision "accept correct reject defer"
        jsonb decided_value "corrections land here"
        varchar actor_type "policy user import process"
        varchar supersedes_decision_id FK "re-decisions append"
    }
    materializations {
        varchar id PK
        varchar document_id FK
        varchar business_object_id FK
        text proposal_set_digest "UNIQUE(document, target_type, target_id)"
    }
    evidence_bindings {
        varchar document_id FK
        varchar business_object_id FK
        binding_kind kind "source or evidence - only evidence projects"
        boolean is_primary
        timestamptz superseded_at "append-only"
    }
    business_objects {
        varchar id PK
        varchar organization_id FK
        varchar object_type "identity spine - no attributes"
    }

Overview of the pipeline core and its seams; the DDL blocks below are authoritative for columns and constraints, and the domain aggregates (orders, freight, shipments, obligations) are drawn in sections 3.4–3.7.

Conventions: every tenant-owned table carries organization_id with composite foreign keys (foreign keys containing both the organization ID and record ID) and RLS (postgreSQL row-level security (RLS), which restricts each query to the customer's rows); global reference data (countries, currencies, ports, carriers, vessels) does not. created_at/updated_at everywhere, omitted below. Types are abbreviated. Critical database constraints that enforce the design are spelled out because they are the design.

3.1 Intake and evidence

deliveries                          -- every arrival, repeat provenance kept; the UNIT of context for reading
  id PK, organization_id, channel (uploaded|email|whatsapp|chat|api|generated),
  actor_type (user|system|counterparty), actor_id NULL,
  contact_id -> contacts NULL,        -- the sender: strongest identity signal, captured here
  conversation_id -> conversations NULL,   -- the (tenant, channel, contact) thread this belongs to
  message_id NULL, in_reply_to NULL, thread_key NULL,   -- provider threading; quoted text is stripped at normalize
  forwarded bool, caption_text NULL,  -- WhatsApp exposes a forwarded flag and withholds the original sender
  requested_context jsonb NULL,       -- shipment hint, forced type, reprocess intent
  idempotency_key, received_at
  UNIQUE (organization_id, channel, idempotency_key)

delivery_artifacts                  -- body and attachments alike; a text-only message has exactly one row
  delivery_id, artifact_id -> artifacts, role (body|attachment|inline), ordinal, filename NULL
  UNIQUE (delivery_id, artifact_id)

conversations                       -- one per (tenant, channel, contact); the identity of the Temporal entity workflow
  id PK, organization_id, channel, contact_id -> contacts,
  temporal_workflow_id, last_inbound_at NULL, last_outbound_at NULL
  UNIQUE (organization_id, channel, contact_id)

dispatches                          -- every OUTBOUND message; the mirror of deliveries
  id PK, organization_id, conversation_id -> conversations, channel,
  kind (pack|reminder|question|receipt|reply|document),
  asks_proposal_ids uuid[] NULL,      -- the open questions this message poses; a reply that answers them is a decision
  template_key NULL, body_ref, artifact_ids uuid[],
  idempotency_key, status (queued|sent|delivered|read|failed), provider_message_id NULL,
  sent_at NULL, failed_reason NULL
  UNIQUE (organization_id, idempotency_key)   -- a retried activity sends once

artifacts                           -- the received file, stored once, addressed by its hash, read-only after write
  id PK, organization_id, content_hash, byte_size, mime_detected,
  storage_key, storage_tier (hot|cold|archive),        -- cost, ops-owned
  retain_until NULL, legal_hold bool                   -- obligation, tenant-owned
  UNIQUE (organization_id, content_hash)

renditions                          -- derived, reproducible; cache not truth
  id PK, artifact_id, kind (page_png|text_layer|region_crop|sheet_file),
  locator, dpi NULL, storage_key

documents                           -- the DECIDED semantic unit; zero self-references
  id PK, organization_id, artifact_id, delivery_id,    -- which arrival produced it
  created_from_proposal_id UNIQUE,    -- the decided classification proposal: a retry cannot create a duplicate document
  generation,                         -- reclassification supersedes the prior set; one current generation per artifact
  artifact_part NULL,                 -- page range or sheet name; NULL = whole artifact
  document_type NOT NULL,             -- born typed: a document exists only once classification has decided
  extraction_variant NULL, current_run_id NULL, deleted_at NULL
  FK (organization_id, document_type) -> document_types
  FK (delivery_id, artifact_id) -> delivery_artifacts (delivery_id, artifact_id)   -- the producing delivery must contain this file

document_families                   -- version chains with enforced identity
  id PK, organization_id, document_type, chain_key
  UNIQUE (organization_id, document_type, chain_key)

document_family_members
  family_id, document_id UNIQUE, revision_no, is_current,
  decided_by_decision_id NULL
  UNIQUE (family_id, revision_no)
  UNIQUE (family_id) WHERE is_current   -- one live head; demotion in the same tx

What this buys, versus the current model: documents loses all four self-referencing foreign keys (columns that point to another row in the same table). Documents split from a larger file are rows with an artifact_part; a merged PDF is an artifact, and no placeholder document row for the unsplit file exists for it. Duplicate handling is a constraint plus a delivery row instead of a unused column and two inconsistent code paths. Retention and tiering (choosing the file's storage class based on cost and retention needs) are separate fields with separate owners.

A message is a delivery whose body is an artifact like any attachment. A WhatsApp text with no file has one delivery_artifacts row of role body; an email with two PDFs has three. Classification runs per artifact but sees every artifact in the delivery: the body's "revised invoice attached, disregard the previous one" is context for the attachment's run, and a body that carries no fact of its own is classified as noise and produces no document. Threading fields are captured at intake so a reply can be matched to its conversation and to the question it answers; quoted text is stripped at normalize so a forwarded message does not re-assert facts already recorded.

3.2 Processing: runs, proposals, decisions

runs                                -- product-facing projection; Temporal owns execution
  id PK, organization_id, document_id NULL, artifact_id NULL,
  purpose (pipeline|reprocess|shadow),
  status (running|waiting_review|completed|failed|superseded),
  code_version, policy_snapshot_id, temporal_workflow_id
  CHECK (num_nonnulls(document_id, artifact_id) = 1)   -- classify/split runs are artifact-level
  UNIQUE (document_id) WHERE status IN ('running','waiting_review')
  UNIQUE (artifact_id) WHERE status IN ('running','waiting_review')
                       AND document_id IS NULL   -- one active file-level run per file

stage_executions                    -- append-only attempts
  id PK, run_id, stage, attempt, input_digest, status, output_ref
  UNIQUE (run_id, stage, input_digest)  -- a retry returns the recorded result instead of re-paying

pipeline_proposals                     -- IMMUTABLE machine observations
  id PK, organization_id, run_id, document_id NULL, stage,
  proposal_type (classification|segment|field|entity_match|entity_create|link|refers_to|assertion|revision|check),
  -- refers_to: which object a MESSAGE is about, with its own threshold. assertion: a statement about an
  -- existing object (attribute, value, modality, effective_at) that materializes an observation (3.5). check: a deterministic validator result
  subject, pass smallint,             -- pass 2 = targeted high-fidelity re-read
  proposed jsonb, normalized jsonb,
  source_ref jsonb,                   -- page+bbox / sheet+cell / text span (artifact, char_start, char_end): what the review UI highlights
  features jsonb,                     -- candidate set, agreement, validator results
  confidence numeric NULL,            -- calibrated, not self-reported
  produced_by jsonb                   -- model, prompt, schema, template, policy versions
  UNIQUE (run_id, proposal_type, subject, pass)

proposal_decisions                  -- APPEND-ONLY; latest per proposal wins
  id PK, organization_id, proposal_id,
  decision (accept|correct|reject|defer), decided_value jsonb NULL,
  actor_type (policy|user|counterparty|import|process), actor_id, rationale NULL,   -- counterparty: a customer's reply, arriving as a delivery, decides a sign-off
  task_id NULL, supersedes_decision_id NULL, decided_at

adjudication_tasks                  -- one envelope, typed handlers per kind
  id PK, organization_id,
  kind (field|entity|association|version|erp_conflict|approval),
  business_object_id NULL, document_id NULL,
  payload jsonb,                      -- schema per kind; handlers own transitions + authz
  status (open|resolved|dismissed), assignee NULL

materializations                    -- idempotent persist record
  id PK, organization_id, run_id, document_id, business_object_id,
  target_type, target_id NULL, proposal_set_digest
  UNIQUE (document_id, target_type, target_id)   -- the instance is part of the key: one message may update two shipments

model_invocations                   -- the billing source of truth
  id PK (deterministic, caller-supplied), organization_id, run_id NULL, purpose,
  model, prompt_version, schema_version, request_fingerprint,
  status (reserved|completed|failed|ambiguous|reconciled),
  provider_generation_id NULL, tokens_in, tokens_out, tokens_cached,
  provider_cost NULL, latency_ms, reserved_at, completed_at NULL

This trio is the core of the design. Proposals do not change: they are what the machine said, and they are the training labels (known expected values used to train or evaluate models). Decisions only add new rows without changing earlier ones: A later decision replacing an earlier one, a second reviewer, a reprocess each add a row and replace while retaining the earlier version, so history survives. Domain rows are written only from accepted proposal (a stored value or relationship proposed by the system, before approval)s, recorded in materializations with an input digest (a hash that uniquely represents the accepted input set), so a redelivered message matches the existing record and makes no change. Corrections carry both the untouched proposal and the decided value, which is what makes each one a labeled example (an input paired with the accepted or corrected expected value). Accuracy per stage is a GROUP BY over these tables; the eval corpus grows as a side effect of operating the product.

3.3 Identity

business_objects                    -- identity spine: id, tenant, type. No attributes.
  id PK, organization_id, object_type

parties        id PK, organization_id, legal_name, country_code, address, ...
party_roles    party_id, role (customer|supplier|forwarder|other), default_terms NULL
contacts       id PK, organization_id, party_id, email, phone, name
products       id PK, organization_id, sku NULL, name, uom, hs_code NULL, ...

identifiers                         -- deterministic tier: matched before anything fuzzy
  id PK, organization_id NULL,        -- NULL = identifier of a global entity
  entity_type, entity_id, id_type (tax_id|sku|unlocode|scac|imo|iata_prefix),
  value
  UNIQUE (COALESCE(organization_id,''), id_type, value)

aliases                             -- learned vocabulary, scoped, provenance-bearing
  id PK, organization_id, entity_type, entity_id,
  counterparty_id -> parties NULL,    -- scoped wherever the counterparty is known
  alias_text, normalized_text,
  source (correction|operator|import|promoted),
  created_from_decision_id NULL, effective daterange NULL
  UNIQUE (organization_id, entity_type, normalized_text, COALESCE(counterparty_id,''))
  -- one meaning per scoped string; a second entity behind the same string is a review task

global reference (platform-owned, no tenant key):
  countries, currencies, ports, carriers, vessels

Resolution order: identifiers, then scoped aliases, then exact matches on standardized names, then finding possible matching records with a calibrated ranker over retained features (a ranking model whose confidence is checked against observed outcomes, using saved matching signals such as identifiers, dates, and parties). A model chooses only between closely matched candidates, and it gets the document's dates. Match, create and correct are three decision kinds; auto-creating master data (authoritative supplier, product, port, carrier, and vessel records) is a records are created only when configuration permits it; extraction alone cannot create them; extraction has no create path. Product resolution reads the alias table, which fixes the single most expensive dead end in the current system: aliases written on every review and read by nothing.

3.4 Commercial and freight

One pattern serves every commercial aggregate (a business object and the records that must change with it, such as an order with its versions and lines): a a permanent top-level record, versions that are not edited after creation, and a field identifying the current version.

orders          id PK, organization_id, side (purchase|sale) NOT NULL,
                buyer_party_id, seller_party_id, status, current_revision_id
order_revisions id PK, order_id, revision_no, terms, currency, incoterm,
                source_document_id NULL, decided_by_decision_id NULL
                UNIQUE (order_id, revision_no)
order_lines     id PK, organization_id, order_revision_id, product_id NULL,
                qty, uom, unit_price, tax, fulfillment_state

commercial_requests, quotes: same pattern, side NOT NULL.

Freight is its own bounded context (a part of the business with its own model and rules) (lifecycle genuinely differs:
comparison, ranking, equipment, routing):
freight_requests, freight_quotes + freight_quote_charges,
freight_comparisons, freight_allocations, freight_bookings (route legs, equipment)
Principal seams to the rest of the model: order-line allocations,
shipment/container allocations, settlement obligations.

3.5 Logistics and tracking

shipments       id PK, organization_id, mode, direction, shipper/consignee/notify
                party ids, status
shipment_legs   id PK, shipment_id, leg_no, from/to (port or location),
                vessel_id NULL, voyage NULL, scheduled/actual dates
handling_units  id PK, organization_id, shipment_id,
                kind (container|package|pallet), identifier, seal NULL

allocations                         -- the three-way-match substrate
  id PK, organization_id, order_line_id, shipment_id,
  handling_unit_id NULL, qty, uom

observations                        -- append-only: what a source asserted about ANY object; nothing here is rewritten
  id PK, organization_id, subject_type (shipment|order|order_line|brand|obligation|...), subject_id,
  attribute, value jsonb, modality (expected|actual),
  effective_at NULL, observed_at, received_at,
  source_class (provider|erp|document|message|operator), source_ref,   -- delivery, document or provider poll
  decision_id NULL,                   -- set when the observation came through the review gate
  payload_hash
  UNIQUE (organization_id, payload_hash)   -- the same report arriving twice is one observation

object_state                        -- deterministic reduction per (subject, attribute), versioned
  subject_type, subject_id, attribute, current_value jsonb, modality,
  reducer_version, derived_from uuid[]      -- the observations that produced this value
  PK (subject_type, subject_id, attribute)

Shipment tracking is the first reducer, and its incident rules stay as versioned code invariants with fixtures: origin port write-once; destination changed only when empty or when the five reroute checks all pass; provider telemetry outranks a document's dates; once an actual milestone is confirmed, later inputs cannot change it; BOL refreshes cannot overwrite operator edits. The table is no longer shipment-shaped because shipments are no longer the only object the world keeps reporting on. A supplier's email pushing a lead time out a week is an observation on an order's promised date; a customer's reply is an observation on a line's sign-off; a forwarded file is an observation on a brand's current version. Each (subject_type, attribute) registers a reducer: a pure function from its observations to a current value, with a precedence ladder over source_class and modality (an actual beats an expected; operator beats message beats document unless a registered rule says otherwise) and a version number, so a rule change can be replayed and diffed. Message-sourced observations reach this table only through the review gate, so a misread email cannot move state without a policy or a person having seen it.

3.6 Settlement

obligations                         -- one aggregate, both directions
  id PK, organization_id, direction (payable|receivable) NOT NULL,
  debtor_party_id, creditor_party_id, currency,
  counterparty_doc_number, posting_state, current_revision_id
  UNIQUE (organization_id, direction, creditor_party_id,
          counterparty_doc_number, doc_class)          -- the chain identity, enforced once

obligation_revisions
  id PK, obligation_id, revision_no, totals, source_document_id,
  ordering_evidence jsonb,            -- manual pin, revision marker/date, generated ts, doc date, receipt time
  decided_by_decision_id NULL, is_current
  UNIQUE (obligation_id) WHERE is_current             -- one live head; demoted in the same tx

obligation_lines   id PK, organization_id, obligation_revision_id,
                   product_id NULL, order_line_id NULL, qty, price, tax

payments           id PK, organization_id, obligation_id,  -- bound to the obligation;
                   amount, date, method, external_ref     -- revision rows hold no payments
payment_schedules  obligation-bound, same reason

adjustments                         -- credit/debit notes are typed adjustments
  id PK, organization_id, obligation_id,
  adjusted_revision_id,               -- the posted revision it legally adjusts
  kind (credit_note|debit_note), lines, external_doc_ref NULL,
  source_document_id NULL

The current invoices.purpose enum dissolves into its three honest ideas: document class stays on the document (a customs invoice is evidence), lifecycle is a status (a proforma is a a preliminary commercial record that does not yet create a payable or receivable linked to the eventual obligation), and a credit note is an adjustment with a sign, bound to both the obligation and the revision it adjusts. The materiality gate and the complete ordering ladder are invariants with fixtures; the two current, inconsistent chain-key implementations become one unique constraint

3.7 Evidence bindings, eventing, ERP, policy

evidence_bindings                   -- documents to business objects; many-to-many
  id PK, organization_id, document_id, business_object_id,
  kind (source|evidence), is_primary bool,
  decision_id NULL, superseded_at NULL, superseded_by NULL
  UNIQUE (document_id, business_object_id, kind) WHERE superseded_at IS NULL
  UNIQUE (document_id, object_type, kind) WHERE is_primary AND superseded_at IS NULL
  -- supersession replaces a specific binding; only evidence bindings project,
  -- and the primary flag decides where a single FK projection needs one winner.
  -- source = provenance ("this BOL created this shipment"); it does not project.

outbox             id PK, topic, event_type, payload, trace, created_at
outbox_deliveries   event_id, subscriber, status, attempts, next_attempt_at
  -- one row per subscriber: a lost intent is detectable and retryable per consumer

erp_commands       id PK, organization_id, integration_id, command, payload,
                   status, attempts                    -- durable command ledger
erp_links          integration_id, entity_type, entity_id, remote_id,
                   watermarks, sync_status (incl. parked conflict)

entity_authority                    -- who owns the fact, per tenant and entity type
  organization_id, entity_type (order|customer|product|invoice|...),
  authority (aradus|erp), integration_id -> integrations NULL, since
  PK (organization_id, entity_type)
  -- default aradus when no ERP is connected. Deliveries, dispatches, artifacts, proposals, decisions,
  -- evidence, policies, brand versions and sign-offs are aradus-owned for every tenant and are not listed here.

policies                            -- the config surface; versioned, effective-dated
  id PK, organization_id NULL, counterparty_id NULL, counterparty_role NULL,
  document_type NULL, layout NULL, channel NULL, field_path NULL,
  policy_type (template|check|matching|autonomy|linkage|versioning|review_routing),
  definition jsonb, version, effective daterange, created_by
  -- autonomy definitions carry a MODE as well as thresholds: auto_when_signals_agree | require_approval | ask_below_similarity
  -- every policy_type declares its runtime reader; a test fails if a surfaced
  -- option has no execution path. Incident rules are NOT here: they are
  -- versioned code invariants; policy may route outcomes to review, not weaken them.

audit_events       business_object_id, actor, action, before_ref, after_ref

4 · Key flows

The trace in full: one invoice, every write, in order

ACME (party pty_9a2) emails invoice INV-4471 for USD 4,200 against shipment shp_113. Every row the pipeline writes, numbered in commit order, showing the values stored in each row. Each row contains IDs linking it to the preceding rows so each write is traceable to the ones before it.

Reads are omitted; the diagram shows writes only, in commit order. Proposals (6, 11–13) and decisions (7, 14, 17) are the only tables written in more than two steps, and both only ever gain rows.

Emailed supplier invoice, happy path

1. Email arrives: upsert artifacts by hash, insert deliveries with the sender's contact_id, open a file-level run, write artifact.received (carrying the delivery, artifact and run IDs) to the outbox. One transaction; no document row yet. A Temporal workflow starts.
2. Classify: model call through the gateway (ledger (a durable record of attempted and completed operations) row reserved, then completed); classification proposals inserted; policy auto-accepts above its threshold. Below it the run pauses for review, as waiting_review and the pending proposal appears in the review queue; the decision's outbox event resumes the workflow.
3. Extract, validate, resolve, link: proposals per field with coordinates, check proposals from deterministic validators, entity-match proposals (the sender's contact resolves the counterparty before any fuzzy matching (approximate name matching rather than exact identifier matching)), ranked link proposals carrying the document's dates. Fields whose checks disagree get a pass-2 targeted re-read; agreement feeds the calibrated confidence.
4. Review: policy decides everything above its per-field thresholds; the rest waits for a person. Decisions append; corrections write decided_value, scoped aliases, and calibration data.
5. Commit: one transaction inserts the materializations row, the obligation (or a revision through the materiality gate), lines with their order-line matches, evidence bindings, and outbox events. Derive projects FK columns from primary evidence bindings.
6. ERP: an erp_commands row per integration, executed idempotently against the customer's books.

Merged PDF, five documents inside

1. Intake creates one delivery, one artifact, and a file-level run, opened before its documents are split (the CHECK on runs allows exactly this case). No document row yet: nothing is known to be a document.
2. Classification proposes each document's type and page range, one proposal per segment with its confidence. Policy decides them; a low-confidence segmentation pauses for review before anything is sliced.
3. Materialize: per decided segment, create a derived file for each approved page range into renditions, insert a documents row with artifact_part and the parent delivery's provenance (the record of where a file or value came from, when it arrived, and who or what produced it), and start a per-document run. A redelivered split re-runs against the same digests and no-ops.
4. Each document proceeds through the happy path independently. A reclassification of the artifact replaces the earlier set while retaining it in history.

Workbooks follow the same flow with sheet names as artifact_part. There is no parent pseudo-document and no fake system actor: The delivery record carries the source information.

Denied segmentation: retry and correction

What happens when the split proposal is wrong. The cost depends entirely on timing: before documents exist, denial is free; after, it is a generation supersession. The gate exists to make the second case rare.

flowchart TD
    P["segmentation proposal<br/>on the file-level run (pass 1)"] --> D{"decision"}
    D -- "reject: wrong, machine<br/>should try again" --> E["escalated re-read:<br/>higher DPI, stronger model"]
    E --> P2["new proposal, pass+1,<br/>same run - prior pass kept"]
    P2 --> D
    D -- "defer" --> W["file run stays<br/>waiting_review"]
    W --> D
    D -- "auto-accept<br/>(policy threshold met)" --> M["materialize: documents born typed<br/>+ per-document runs"]
    D -- "correct: the reviewer's boundaries<br/>become the decided_value" --> M
    M --> OK(["pipeline continues<br/>per document"])
    OK -.-> L["split found wrong LATER,<br/>documents already exist"]
    L --> R["explicit resegment:<br/>new file-level run"]
    R --> NP["new segmentation proposal"]
    NP --> ND{"decision"}
    ND -- "accept / correct" --> G["generation+1 documents,<br/>born typed from the corrected split"]
    G --> SUP["old generation superseded, not deleted:<br/>runs, proposals, decisions, bindings kept;<br/>materialized records reconciled by supersession"]

Reject cannot loop unbounded: each escalated pass produces a new proposal that costs a fresh decision, and correct is the terminal fallback - the reviewer states the truth and documents are created from it. Every wrong guess stays on record as a labeled example.

Duplicate arrival, and re-upload with intent

If the resent file has the same hash, reuse the existing file record: a new deliveries row records the arrival (sender, channel, time) and nothing reprocesses - existing documents keep their original producing delivery and are not retargeted. A manual re-upload whose request metadata states what the operator wants also reuses the stored file: a reprocess request opens new runs on the existing documents; only an explicit reclassify or resegment creates a replacement document generation. Content is stored once in both cases; the difference between "we received it again" and "the operator wants it re-read" is a field on the delivery row rather than a code path.

Revised bill

The new document lands in the same document_family by chain key (the fields used to decide that two documents are versions of the same bill). The materiality gate compares with the current version: an empty or trivial diff returns the head unchanged, so a re-upload cannot reset an approved bill. A material diff creates a revision proposal; the decision (policy or human, per the tenant's versioning policy) inserts an obligation_revisions row, demotes the old head in the same transaction, and payments stay bound to the obligation. An older document is stored as a non-current revision without changing the current version or its payments.

BOL creates a shipment

1. Extraction and resolution produce proposals for the BL number, parties, ports, containers, and an entity_create proposal for the shipment itself, since no matching shipment exists.
2. On acceptance, commit create,s the shipment, its legs and handling units (trackable containers, packages, or pallets) in one transaction, with the materializations digest guarding redelivery. A container number that fails its check digit (a validation digit used to detect mistyped container numbers) materializes the unit with a null identifier and a pending field task; keep the suspected number only as an unapproved proposal; do not use it for tracking.
3. An evidence binding of kind source records that this document created this shipment. It does not populate any derived shipment field; a duplicate BOL arriving later gets an evidence binding only.
4. Tracking initialization appends the first observations; the reducer computes initial state; from here provider polling (regularly requesting tracking updates) and the BOL disagreeing about dates is the reducer's problem, under its versioned precedence rules (versioned rules deciding which source wins when dates conflict).

Ambiguous shipment link

Two candidate shipments for one certificate: The proposed match stores both candidate shipments and the evidence used to compare them including the document's own dates, so a container reused eight months apart resolves without a person. A genuine tie leaves the link proposal pending; the human review task shows both candidates with evidence; the decision creates the evidence binding, and derive updates the projections. A later manual re-link supersedes that specific binding and re-derives. No scheduled retry calls and charges the model again while review is pending while the task sits open.

A correction that teaches

1. An operator corrects the seller on a pending entity-match proposal: the decision row records decided_value beside the untouched proposal, the actor, and the task.
2. The same transaction writes a scoped alias ("this counterparty's 'ACME TRDG LLC.' means party X") with created_from_decision_id, subject to the an alias may identify only one record; a conflicting alias opens a review task; a collision with an existing alias opens an ambiguity task instead of silently overwriting.
3. The proposal-decision pair becomes a labeled example: it joins the calibration set (examples used to measure confidence accuracy) for that field and counterparty, and the few-shot retrieval (selecting relevant worked examples for each model prompt) pool (examples eligible for inclusion in later prompts) for the next document from that sender, with evaluation cases kept separate.
4. The next invoice from this supplier matches the seller exactly from the saved alias before calling a model, before any model runs.

Reprocess and change type

Reprocess opens a new run; the old run and its proposals stay, marked superseded. New proposals are decided (prior human decisions auto-apply where subject and value match; the rest re-enter review), commit re-materializes through the same digests, updating rows in place where the target exists, and evidence bindings are superseded and recreated rather than deleted. A change of type does not update document_type in place: it is a human-decided classification proposal that creates a replacement document generation, supersedes the old one, and reconciles its evidence bindings and materialized records. Nothing is hard-deleted, so a reprocess cannot destroy corrections, and the a safeguard requiring approval when changing the type would leave a shipment without its creating document is a policy rule: a type change that would orphan a shipment routes to an approval task instead of being forbidden in code.

ERP two-sided conflict

1. The inbound pull detects that both Aradus and the ERP changed the same record since the last sync. The erp_links row is marked as a conflict; the conflict remains paused until a person resolves it; routine synchronization must not resume it, since the park's own bookkeeping must not un-park it on the next sweep.
2. An adjudication task of kind erp_conflict opens with both versions and the field diff. No writes flow in either direction while it is open.
3. The operator picks theirs, ours, or a merge. The decision releases the conflict, and the outcome becomes an erp_commands row (ours) or an inbound patch (theirs), each idempotent under the customer's compatibility contract (customer-specific rules that preserve the ERP behavior their accounting processes rely on).

Text-only message about an existing object

1. A carrier emails "MSC Adriana will discharge at Khalifa Port, ETA 27 March." The delivery has one body artifact; classification proposes correspondence and the document is born typed.
2. The run produces a refers_to proposal (the vessel name and container number in the text resolve to one open shipment, 0.97) and two assertion proposals: destination port = Khalifa (actual) and ETA = 27 March (expected), each with a text-span source_ref.
3. Policy for this tenant trusts carrier senders about routing, so the port assertion auto-accepts and materializes an observation. The ETA assertion auto-accepts as expected, and a notification fires because the slip exceeds the tenant's two-day rule. Both materializations key on the shipment instance, so a second email about a second container updates a second shipment without collision.
4. When the provider feed later reports the actual berth, that observation outranks the expected one and the state moves. The email's observation stays on record as what the carrier said.

A person messages first

A contact sends "cancel line 3 of yesterday's order" with nothing pending. Intake does SignalWithStart on conversation-{tenant}-{channel}-{contact}. The conversation workflow's interpret activity classifies the message as an instruction about an object, resolves "yesterday's order" against the contact's recent deliveries, and produces an assertion proposal (line 3 status = cancelled) that goes through the same gate as any other. Had the message answered an open question, the conversation would have signalled the workflow that asked. Had it been a question, an activity would have read current state and drafted a reply as a dispatch. Every turn is a delivery or a dispatch row; the workflow holds only the open questions and pointers to the last few turns, and continues as new after a bounded number of turns.

5 · Deep dives

Confidence and calibration

The gate does not read a model's self-assessment. Inputs to the calibrated score: cross-pass agreement (pass 2 uses a different model or rendering, since two correlated passes are weak evidence), deterministic validator results (results from fixed validation rules), resolver features (saved signals used when matching records), and the counterparty's track record. The calibrator maps raw scores to observed correction rates (adjusts model scores using how often similarly scored values were actually corrected) per document type, field and layout. Its training data has three sources by design: gold cases (manually verified examples with known correct answers), production corrections, and random audits of auto-accepted work, because a dataset containing only corrections is biased toward cases that were already sent to reviewers. Hierarchical priors cover cold starts (when a new customer or layout has little data, begin with broader field- and document-type statistics, then adapt as local evidence accumulates); retrieval examples are kept out of evaluation sets (examples shown to the model must not also be used to measure its accuracy).

Policy versus invariant

Configuration is applied from platform defaults down to field-specific rules (platform → tenant → counterparty and role → document type → layout → channel → field). It holds statements of business fact: templates with field-to-column maps, checks with tolerances and severities, matching strategy order, linkage conventions, materiality rules, autonomy grants. Use centrally managed thresholds based on observed error rates; customer-specific thresholds would overfit small datasets. Incident-derived correctness rules are not policy at all: the ordering ladder, tracking precedence and ERP idempotency are versioned code with fixtures (rules implemented in versioned code and protected by fixed regression examples), and policy can only choose whether an outcome routes to review.

Idempotency where money moves

Three ledgers make repeated delivery harmless (three durable operation records prevent retries from repeating billable or financial effects). model_invocations: a call is reserved under a deterministic ID before dispatch (record the call under a repeatable ID before sending it; after a crash, confirm the provider's result instead of charging again; billing uses completed calls rather than message deliveries), so a crash after the provider answered is reconciled, not re-billed, and billing reads this table rather than queue traffic. materializations: A repeated commit matches the existing input hash and makes no change. erp_commands: every push is a durable command with attempts (a stored ERP request with a retry history), and the customer-facing semantics (external ID schemes, for products first created in the ERP, treat an exact SKU match as the same product, case-insensitive number matching within tenant and direction, draft-only posting, already-posted no-ops, line replacement, conflicts that stay paused until a person resolves them) are a per-customer compatibility contract, changed only after certification against that customer's system.

Why Python survives, and shrunk to what

Model calls are just HTTP with a JSON schema; Go could make them. Python stays for two narrow reasons: PDF and image tooling (rasterization, DPI control, region crops for pass-2 re-reads (rendering PDF pages as images at a controlled resolution and rereading selected page regions)) where the Python ecosystem is years ahead, and the model-iteration loop, since few-shot assembly, eval tooling and DSPy-class prompt optimization are Python-first (python has stronger tools for selecting prompt examples, evaluating results, and automatically improving prompts) and that loop is where the accuracy advantage accumulates. The discipline that makes the split cheap: the gateway is stateless, has no database credentials, and every schema is defined once in JSON Schema and generated into both runtimes. The current system's Pydantic-plus-Go-struct duplication, which is why templates cannot be authored today, is the architecture prevents this duplication.

Messages as input: six changes

Documents arrive with their own identifiers and a fixed shape; messages do not. Six changes make a message a first-class input rather than an attachment carrier. Bodies are artifacts and deliveries carry threading (3.1). Linkage is its own proposal type, refers_to, with its own threshold, because for a message the hard question is which object it is about rather than what it says (3.2). Evidence anchors include a text span, so a review shows the quoted words rather than a page (3.2). Statements about existing objects are assertion proposals that materialize observations, and the reducer covers every subject type rather than only shipments (3.5). Materialization is keyed by target instance, so one message can update several objects of one type (3.2). Extraction schemas for correspondence are assertion-shaped: a variable-length list of (subject, attribute, value, modality, effective time) rather than fixed fields, declared as a document type whose schema allows a list. The delivery is the unit of context for all six: one message, one body, N attachments, read together.

Who owns which fact

Two kinds of record are easy to confuse. The ERP records business facts: an order, an invoice, a customer. Aradus records how a fact came to be: the message it arrived in, what the machine proposed, what a person decided, the evidence, and the rule that came out of it. The second kind is ours for every tenant and is the memory that makes the system better per customer; no ERP holds it. The first kind is owned per tenant and per entity type through entity_authority. When an ERP is connected it originates and corrects the fact, our row mirrors it with the remote id, and an accepted proposal becomes an erp_commands push rather than a local write. When no ERP is connected, we are the record, edits happen here, and an ERP added later is a downstream target. Same tables in both modes; the flag decides who may originate a fact and which way sync runs. The consequence is accepted: for a tenant with no ERP, being the record of their orders means being their ERP.

Conversations are workflows

Headless and conversational are the same step at two points on the confidence scale. A clean spreadsheet order and a photo of a handwritten one both produce a sales-order proposal; one clears the tenant's threshold and materializes, the other becomes a question. The question is a dispatch to the person on their channel, the reply is a delivery, and the reply is read as a decision on the open proposal. So a chat is a workflow: state is the set of open questions and pointers to recent turns, an inbound message is a signal, a model call is an activity (one to phrase the question, one to read the answer), an outbound message is a dispatch. There is no separate agent; the workflow is the agent, and headless is the same code with the question step skipped. Messages a person starts land in a per-contact entity workflow keyed conversation-{tenant}-{channel}-{contact}, created by SignalWithStart, which routes each message as an answer to an open question, an instruction about an object, or a question to be answered from state. Its id gives the tenant discrimination the current WhatsApp routing lacks. Automated feeds stay on the observation path and are kept out of conversations, because a workflow processes its signals one at a time.

Runtime: self-hosted Temporal, control flow in code

Temporal is self-hosted on the existing Kubernetes cluster for data residency, with the CNPG Postgres operator as persistence. It is a durable execution runtime for code and offers no visual authoring layer. The current system's Engine B bundles two things: durable execution, done by hand with rows, a poller and a seven-day stale sweep, and a graph builder operators can edit. The greenfield keeps neither. Control flow is code, as Temporal workflows, versioned like code: match, send, wait, remind, convert is the same shape for every tenant. Everything a tenant varies is policy data read by the workflow at each step: who to send to, how often to remind, which threshold auto-accepts, whether a marked-up screenshot counts, which mode a replacement runs in (auto_when_signals_agree, require_approval, ask_below_similarity). The operator-facing surface is a rulebook rather than a canvas. Model calls live only in activities; workflows carry ids rather than customer data; long chases continue as new; and worker versioning is decided before the first workflow runs for weeks, since a deploy that changes a running workflow's logic without it fails on replay.

6 · Build plan at 100% feature parity

Parity is defined in user terms before anything is built: A machine-testable list of every user-visible capability (channels, duplicate rules, file types, document types classified as domain-producing or evidence-only, fields, resolutions, mutations, links, reviews, statuses, notifications, metering, tracking rules, ERP operations, retry and give-up paths, permissions, audit). Incident rules become characterization tests (tests that capture the current behavior - including incident cases - so the replacement cannot regress it) carrying their original incident examples. Table shapes, the the two current, inconsistent ways of identifying versions of the same bill and creating business records without committing all related changes atomically are details of the current implementation, not required user behavior, not parity.

First build one minimal end-to-end workflow and its shared foundations; then add complete document types one at a time. The spine is the honest name for the first deliverable, because that is where schedule risk lives: deliveries, artifacts, decomposition, runs, proposals, decisions, policy, the invocation ledger (the durable record of every model call and its cost), a review UI that shows the source page and the highlighted value, commit, outbox, audit, supporting one intentionally minimal end-to-end document flow. Then deliver each document type end to end, including its review UI, since semantic errors show up in review UX, not backend tests: supplier bills first (revisions, materiality, obligations, ERP posting), then BOL (shipment creation, tracking reduction) and customer PO (buyer layouts, allocations), then remaining domain-producing families, then the the remaining document types that provide supporting evidence but create no business records as configuration. The eval corpus and CI gates (automated checks that must pass before a change can merge or deploy) exist from the spine onward; each slice adds its gold cases before its model calls ship.

Migration uses the the allowed maintenance-window duration honestly. One global maintenance window (per-tenant windows on a shared database are would require the old and new systems to operate simultaneously): pause intake, tracking, workflows and outbound connectors (integrations that send updates to ERPs, tracking providers, or notification systems); snapshot; run migration scripts previously tested against production-like snapshots; validate; start the new system with keep outbound ERP updates paused; release connectors customer by customer after read-only verification against their live system. History that cannot honestly carry new-model provenance (extraction JSON without coordinates or model versions, overwritten tracking rows) imports as tagged legacy baselines and synthetic observations (import old values as explicitly marked starting data, and represent overwritten tracking history as generated records rather than presenting them as original source evidence) rather than invented provenance. Every non-deterministic case lands in an exception ledger (a queue of migration cases requiring human review) for human adjudication. Legacy IDs persist in a mapping registry (a table mapping legacy IDs to new-system IDs). Rollback is defined, not assumed: either writes stay blocked until the commitment point (the moment rollback stops being simple; any change after the switchover is recorded so recovery can restore to a defined point), or every post-cutover mutation is journaled with a stated recovery point.

Produced 4 Sep 2026 against branch next; revised 9 Sep 2026 with messages as input, authority per tenant and entity, conversations as workflows, and the self-hosted Temporal runtime. Companion pages: Order Confirmation Design and Aradus Field Guide. Earlier companions: Document Pipeline ERD (the brownfield V0/V1 path) and Pipeline Write Paths.