Greenfield Pipeline Design

1 · Problem and requirements

Every company that moves goods runs on two kinds of information. The record: an order was placed, an invoice issued, a container booked. And the traffic: the emails, PDFs, WhatsApp messages, phone calls and portal screens through which the record gets made and kept current. ERPs hold the record. The traffic is read by people, one message at a time, and typed into the record by hand. Aradus does the traffic. It reads what arrives, decides what it means, does the next thing or asks a person, and remembers how it decided so it does not ask twice. The ERP, or the spreadsheet where there is no ERP, stays the record and is written last.

Four layers, and the design is those four done properly. Edge: the doors in and out (email, WhatsApp, voice, uploads, provider feeds). Reading: turning pixels and text into candidate facts with a confidence and the evidence behind them. Deciding: a policy or a person turns a candidate into a fact and acts on it. Memory: what was decided, on what evidence, and the rule that came out of it. The last layer is the one no ERP has and the reason a customer stays.

Four kinds of record stay separate throughout. A delivery is how something arrived and from whom. An artifact is the bytes, stored once. A document is one business document inside an artifact, and it exists only once classification has decided what it is. A business object is the shipment, order, bill or brand version the documents are about. Most of the current system's incidents come from collapsing two of these into one row.

Functional requirements

• Edge. Ingest through upload, email, WhatsApp, chat, API and provider feeds, message bodies and attachments alike; store one copy per file and record every arrival with sender, channel and time. Send on the same channels through idempotent dispatches, with WhatsApp's template rule respected.
• Reading. Classify, split, extract, normalize and validate about 25 document types, nine of which create or update business records; read statements in message bodies as assertions about existing objects; match names and codes to parties, products, ports, carriers and vessels through scoped aliases; link documents and messages to shipments and orders, and send genuine ambiguity to review.
• Deciding. Version supplier bills with a materiality gate and a fixed tie-break; confirm artwork per order line before a supplier PO; track shipments by bill of lading with provider precedence and a vessel fallback; three-way match; freight requests, quotes and bookings; generate packing lists and commercial invoices from a loading list; answer or route what people send; notify on state changes and in digests; sync with Odoo and Business Central without changing the accounting behaviour customers rely on.
• Memory. Every accepted value traceable to its proposal, evidence, policy and decision with the actor recorded; every correction a labeled example; every intervention offers to become a rule; per-tenant and per-counterparty policy for thresholds, contacts, modes and templates; the usage and cost of every model call, failed ones included.

Non-functional requirements

• Multi-tenant with row-level security: a bug in one tenant's query cannot read another's rows.
• At-least-once delivery everywhere and every effect idempotent; a billable model call is not repeated on redelivery; a retried send goes out once.
• Conversations and chases survive restarts and deploys: a reminder due in two days fires in two days.
• Accuracy measurable per stage, tenant and document type, and attributable to a model, prompt or dataset change.
• Autonomy configurable by field, counterparty and document type, gated on calibrated confidence rather than model self-assessment, with a mode as well as a threshold.
• Evidence retained per tenant policy with legal hold; a customs dispute runs years.
• Onboarding a new counterparty format or a new tenant rule costs configuration, not engineering.

Scale honesty

This is a correctness problem rather than a throughput one. Production creates on the order of one shipment a day and tens of orders a month per tenant; at a hundred times that, one Postgres primary handles every table here without partitioning. The scarce resources are model spend, human review minutes and correctness under retries, and the design optimizes for those.

2 · Architecture

Five components, and the reason each earns its place:

• One modular Go codebase deployed as API and worker pools. No microservices: the transactional boundaries around documents, obligations, shipments and reviews matter more than deployment independence at this scale.
• Self-hosted Temporal on the existing Kubernetes cluster handles execution: leases, retries, timers, heartbeats, fan-out and fan-in, review waits, conversation signals, cancellation, and continuity across deployments. Postgres stays the only business truth; the UI and API read run status from Postgres projections, not from Temporal. Self-hosting is a data-residency decision and carries an operations cost: four services plus persistence, and someone on call who has run it.
• Control flow is code and tenant variation is policy. Workflows are written and versioned like code; every value a tenant changes (recipients, reminder intervals, thresholds, modes) is a policy row read at each step. There is no operator-editable graph builder; the operator surface is a rulebook. Conversations with people are workflows too, one per contact and channel, so headless and chat-driven steps share one code path (section 6).
• A Python media and model gateway, deliberately shrunk: PDF and image rasterization (PyMuPDF has no Go equal), provider adapters and their quirks, and the the process for testing and improving extraction prompts and examples (few-shot assembly, future future automated prompt optimizers). It is stateless, has no database credentials, and schemas are defined once in JSON Schema and used to generate matching Go and Python types, so the the Go and Python schema definitions becoming inconsistent the current system suffers cannot recur. If the PDF tooling gap ever closes, this component can move into the Go application; nothing else depends on its language.
• PostgreSQL, one primary: Authoritative stored data, row-level security provides a second enforcement layer between customers' data, and an append-only domain event log written by activities alongside each fact: a record for audit and analytics rather than a transport (section 6, consistency without an outbox).
• S3: the original files (written once, then read-only) plus derived files such as page images and crops, which are regenerable copies rather than authoritative source data.

3 · Data model

erDiagram
    conversations ||--o{ deliveries : "inbound - one thread per tenant, channel, contact"
    conversations ||--o{ dispatches : "outbound - the mirror of deliveries"
    deliveries ||--o{ delivery_artifacts : "body and attachments alike"
    delivery_artifacts }o--|| artifacts : "many arrivals, one stored file"
    artifacts ||--o{ renditions : "derived page images and crops"
    artifacts ||--o{ documents : "artifact_id"
    deliveries ||--o{ documents : "delivery_id - the producing arrival, same file enforced"
    pipeline_proposals |o--o{ documents : "created_from_proposal_id UNIQUE - born typed"
    document_families ||--o{ document_family_members : "family_id"
    documents ||--o| document_family_members : "document_id UNIQUE"
    artifacts |o--o{ pipeline_runs : "file-level runs (classification)"
    documents |o--o{ pipeline_runs : "per-document runs - CHECK exactly one of"
    pipeline_runs ||--o{ pipeline_proposals : "run_id"
    pipeline_runs ||--o{ model_invocations : "reserved before dispatch - billing truth"
    pipeline_proposals ||--o{ proposal_decisions : "append-only verdicts - latest wins"
    proposal_decisions |o--o{ adjudication_tasks : "task_id when a person decided"
    proposal_decisions |o--o{ entity_aliases : "a correction writes a scoped alias"
    proposal_decisions |o--o{ evidence_bindings : "an accepted link creates the edge"
    documents ||--o{ evidence_bindings : "document_id"
    evidence_bindings }o--|| business_objects : "typed target - an enforced FK"
    pipeline_runs ||--o{ materializations : "accepted set digest - idempotent persist"
    materializations }o--|| business_objects : "the domain record it created"
    business_objects ||..o{ domain_aggregates : "orders freight shipments obligations..."
    organizations ||--o{ pipeline_policies : "the config cascade"

    deliveries {
        varchar id PK
        varchar organization_id FK
        intake_channel channel
        varchar contact_id FK "sender identity"
        varchar conversation_id FK "thread"
        varchar idempotency_key UK
    }
    artifacts {
        varchar id PK
        varchar organization_id FK
        text content_hash "UNIQUE per org"
        text storage_tier "cost"
        timestamptz retain_until "obligation"
    }
    documents {
        varchar id PK
        varchar artifact_id FK "NOT NULL"
        varchar delivery_id FK
        varchar created_from_proposal_id FK "UNIQUE - retry-safe identity"
        integer generation "resegment supersedes the set"
        text artifact_part "pages or sheet"
        varchar document_type "NOT NULL - born typed"
    }
    pipeline_runs {
        varchar id PK
        varchar document_id FK "nullable"
        varchar artifact_id FK "nullable - CHECK one of"
        run_status status "running waiting_review completed failed superseded"
        varchar policy_snapshot_id
    }
    pipeline_proposals {
        varchar id PK
        varchar run_id FK
        proposal_type proposal_type "classification field assertion refers_to entity_match link revision"
        varchar subject
        smallint pass "UNIQUE run+type+subject+pass"
        jsonb proposed
        jsonb source_ref "page bbox - what review highlights"
        numeric confidence "calibrated"
        jsonb produced_by "model prompt schema policy versions"
    }
    proposal_decisions {
        varchar id PK
        varchar proposal_id FK
        decision decision "accept correct reject defer"
        jsonb decided_value "corrections land here"
        varchar actor_type "policy user import process"
        varchar supersedes_decision_id FK "re-decisions append"
    }
    materializations {
        varchar id PK
        varchar document_id FK
        varchar business_object_id FK
        text proposal_set_digest "UNIQUE(document, target_type, target_id)"
    }
    evidence_bindings {
        varchar document_id FK
        varchar business_object_id FK
        binding_kind kind "source or evidence - only evidence projects"
        boolean is_primary
        timestamptz superseded_at "append-only"
    }
    business_objects {
        varchar id PK
        varchar organization_id FK
        varchar object_type "identity spine - no attributes"
    }

Overview of the pipeline core and its seams; the DDL blocks below are authoritative for columns and constraints, and the domain aggregates (orders, freight, shipments, obligations) are drawn in sections 3.4–3.7.

Conventions: every tenant-owned table carries organization_id with composite foreign keys (foreign keys containing both the organization ID and record ID) and RLS (postgreSQL row-level security (RLS), which restricts each query to the customer's rows); global reference data (countries, currencies, ports, carriers, vessels) does not. created_at/updated_at everywhere, omitted below. Types are abbreviated. Critical database constraints that enforce the design are spelled out because they are the design.

3.1 Intake and evidence

deliveries                          -- every arrival, repeat provenance kept; the UNIT of context for reading
  id PK, organization_id, channel (uploaded|email|whatsapp|chat|api|generated),
  actor_type (user|system|counterparty), actor_id NULL,
  contact_id -> contacts NULL,        -- the sender: strongest identity signal, captured here
  conversation_id -> conversations NULL,   -- the (tenant, channel, contact) thread this belongs to
  message_id NULL, in_reply_to NULL, thread_key NULL,   -- provider threading; quoted text is stripped at normalize
  forwarded bool, caption_text NULL,  -- WhatsApp exposes a forwarded flag and withholds the original sender
  requested_context jsonb NULL,       -- shipment hint, forced type, reprocess intent
  idempotency_key, received_at
  UNIQUE (organization_id, channel, idempotency_key)

delivery_artifacts                  -- body and attachments alike; a text-only message has exactly one row
  delivery_id, artifact_id -> artifacts, role (body|attachment|inline), ordinal, filename NULL
  UNIQUE (delivery_id, artifact_id)

conversations                       -- one per (tenant, channel, contact); the identity of the Temporal entity workflow
  id PK, organization_id, channel, contact_id -> contacts,
  temporal_workflow_id, last_inbound_at NULL, last_outbound_at NULL
  UNIQUE (organization_id, channel, contact_id)

dispatches                          -- every OUTBOUND message; the mirror of deliveries
  id PK, organization_id, conversation_id -> conversations, channel,
  kind (pack|reminder|question|receipt|reply|document),
  asks_proposal_ids uuid[] NULL,      -- the open questions this message poses; a reply that answers them is a decision
  template_key NULL, body_ref, artifact_ids uuid[],
  idempotency_key, status (queued|sent|delivered|read|failed), provider_message_id NULL,
  sent_at NULL, failed_reason NULL
  UNIQUE (organization_id, idempotency_key)   -- a retried activity sends once

artifacts                           -- the received file, stored once, addressed by its hash, read-only after write
  id PK, organization_id, content_hash, byte_size, mime_detected,
  storage_key, storage_tier (hot|cold|archive),        -- cost, ops-owned
  retain_until NULL, legal_hold bool                   -- obligation, tenant-owned
  UNIQUE (organization_id, content_hash)

renditions                          -- derived, reproducible; cache not truth
  id PK, artifact_id, kind (page_png|text_layer|region_crop|sheet_file),
  locator, dpi NULL, storage_key

documents                           -- the DECIDED semantic unit; zero self-references
  id PK, organization_id, artifact_id, delivery_id,    -- which arrival produced it
  created_from_proposal_id UNIQUE,    -- the decided classification proposal: a retry cannot create a duplicate document
  generation,                         -- reclassification supersedes the prior set; one current generation per artifact
  artifact_part NULL,                 -- page range or sheet name; NULL = whole artifact
  document_type NOT NULL,             -- born typed: a document exists only once classification has decided
  extraction_variant NULL, current_run_id NULL, deleted_at NULL
  FK (organization_id, document_type) -> document_types
  FK (delivery_id, artifact_id) -> delivery_artifacts (delivery_id, artifact_id)   -- the producing delivery must contain this file

document_families                   -- version chains with enforced identity
  id PK, organization_id, document_type, chain_key
  UNIQUE (organization_id, document_type, chain_key)

document_family_members
  family_id, document_id UNIQUE, revision_no, is_current,
  decided_by_decision_id NULL
  UNIQUE (family_id, revision_no)
  UNIQUE (family_id) WHERE is_current   -- one live head; demotion in the same tx

What this buys, versus the current model: documents loses all four self-referencing foreign keys (columns that point to another row in the same table). Documents split from a larger file are rows with an artifact_part; a merged PDF is an artifact, and no placeholder document row for the unsplit file exists for it. Duplicate handling is a constraint plus a delivery row instead of a unused column and two inconsistent code paths. Retention and tiering are separate fields with separate owners.

A message is a delivery whose body is an artifact like any attachment. A WhatsApp text with no file has one delivery_artifacts row of role body; an email with two PDFs has three. Classification runs per artifact but sees every artifact in the delivery: the body's "revised invoice attached, disregard the previous one" is context for the attachment's run, and a body that carries no fact of its own is classified as noise and produces no document. Threading fields are captured at intake so a reply can be matched to its conversation and to the question it answers; quoted text is stripped at normalize so a forwarded message does not re-assert facts already recorded.

3.2 Processing: runs, proposals, decisions

runs                                -- product-facing projection; Temporal owns execution
  id PK, organization_id, document_id NULL, artifact_id NULL,
  purpose (pipeline|reprocess|shadow),
  status (running|waiting_review|completed|failed|superseded),
  code_version, policy_snapshot_id, temporal_workflow_id
  CHECK (num_nonnulls(document_id, artifact_id) = 1)   -- classify/split runs are artifact-level
  UNIQUE (document_id) WHERE status IN ('running','waiting_review')
  UNIQUE (artifact_id) WHERE status IN ('running','waiting_review')
                       AND document_id IS NULL   -- one active file-level run per file

stage_executions                    -- append-only attempts
  id PK, run_id, stage, attempt, input_digest, status, output_ref
  UNIQUE (run_id, stage, input_digest)  -- a retry returns the recorded result instead of re-paying

pipeline_proposals                     -- IMMUTABLE machine observations
  id PK, organization_id, run_id, document_id NULL, stage,
  proposal_type (classification|segment|field|entity_match|entity_create|link|refers_to|assertion|revision|check),
  -- refers_to: which object a MESSAGE is about, with its own threshold. assertion: a statement about an
  -- existing object (attribute, value, modality, effective_at) that materializes an observation (3.5). check: a deterministic validator result
  subject, pass smallint,             -- pass 2 = targeted high-fidelity re-read
  proposed jsonb, normalized jsonb,
  source_ref jsonb,                   -- page+bbox / sheet+cell / text span (artifact, char_start, char_end): what the review UI highlights
  features jsonb,                     -- candidate set, agreement, validator results
  confidence numeric NULL,            -- calibrated, not self-reported
  produced_by jsonb                   -- model, prompt, schema, template, policy versions
  UNIQUE (run_id, proposal_type, subject, pass)

proposal_decisions                  -- APPEND-ONLY; latest per proposal wins
  id PK, organization_id, proposal_id,
  decision (accept|correct|reject|defer), decided_value jsonb NULL,
  actor_type (policy|user|counterparty|import|process), actor_id, rationale NULL,   -- counterparty: a customer's reply, arriving as a delivery, decides a sign-off
  task_id NULL, supersedes_decision_id NULL, decided_at

adjudication_tasks                  -- one envelope, typed handlers per kind
  id PK, organization_id,
  kind (field|entity|association|version|erp_conflict|approval),
  business_object_id NULL, document_id NULL,
  payload jsonb,                      -- schema per kind; handlers own transitions + authz
  status (open|resolved|dismissed), assignee NULL

materializations                    -- idempotent persist record
  id PK, organization_id, run_id, document_id, business_object_id,
  target_type, target_id NULL, proposal_set_digest
  UNIQUE (document_id, target_type, target_id)   -- the instance is part of the key: one message may update two shipments

model_invocations                   -- the billing source of truth
  id PK (deterministic, caller-supplied), organization_id, run_id NULL, purpose,
  model, prompt_version, schema_version, request_fingerprint,
  status (reserved|completed|failed|ambiguous|reconciled),
  provider_generation_id NULL, tokens_in, tokens_out, tokens_cached,
  provider_cost NULL, latency_ms, reserved_at, completed_at NULL

This trio is the core of the design. Proposals do not change: they are what the machine said, and they are the training labels. Decisions only add new rows without changing earlier ones: A later decision replacing an earlier one, a second reviewer, a reprocess each add a row and replace while retaining the earlier version, so history survives. Domain rows are written only from accepted proposals, recorded in materializations with an input digest, so a redelivered message matches the existing record and makes no change. Corrections carry both the untouched proposal and the decided value, which is what makes each one a labeled example. Accuracy per stage is a GROUP BY over these tables; the eval corpus grows as a side effect of operating the product.

3.3 Identity

business_objects                    -- identity spine: id, tenant, type. No attributes.
  id PK, organization_id, object_type

organizations                       -- the TENANT: the account that logs in, has members, roles, a plan, integrations. Unchanged.
  id PK, name, slug, ...                -- counterparty rows leave this table; an organization without members is a bug

partners                            -- everyone a tenant deals with, one row per (tenant, legal entity); the tenant itself is a partner row too
  id PK, organization_id, legal_name, country_code, address,
  as_organization_id -> organizations NULL   -- set when this partner is also an Aradus tenant; two views, one link
partner_roles  partner_id, role (customer|supplier|carrier|forwarder|notify|self), default_terms NULL
contacts       id PK, organization_id, partner_id -> partners, name, title NULL
contact_channels                    -- a person has N addresses; a conversation resolves an address to one contact
  id PK, organization_id, contact_id -> contacts, channel (email|whatsapp|phone),
  address, verified_at NULL, opt_in_at NULL
  UNIQUE (organization_id, channel, address)
products       id PK, organization_id, sku NULL, name, uom, hs_code NULL, ...
components, bill_of_materials       -- move unchanged; nothing here changes what they mean

identifiers                         -- deterministic tier: matched before anything fuzzy
  id PK, organization_id NULL,        -- NULL = identifier of a global entity
  entity_type, entity_id, id_type (tax_id|sku|unlocode|scac|imo|iata_prefix),
  value
  UNIQUE (COALESCE(organization_id,''), id_type, value)

aliases                             -- learned vocabulary, scoped, provenance-bearing
  id PK, organization_id, entity_type, entity_id,
  counterparty_id -> partners NULL,   -- scoped wherever the counterparty is known
  alias_text, normalized_text,
  source (correction|operator|import|promoted),
  created_from_decision_id NULL, effective daterange NULL
  UNIQUE (organization_id, entity_type, normalized_text, COALESCE(counterparty_id,''))
  -- one meaning per scoped string; a second entity behind the same string is a review task

global reference (platform-owned, no tenant key):
  countries, currencies, ports, carriers, vessels

Resolution order: identifiers, then scoped aliases, then exact matches on standardized names, then finding possible matching records with a calibrated ranker over retained features. A model chooses only between closely matched candidates, and it gets the document's dates. Match, create and correct are three decision kinds; auto-creating master data (authoritative supplier, product, port, carrier, and vessel records) is a records are created only when configuration permits it; extraction alone cannot create them; extraction has no create path. Product resolution reads the alias table, which fixes the single most expensive dead end in the current system: aliases written on every review and read by nothing.

Two things today's organizations table does are pulled apart. The tenant, the account that logs in, keeps the table and its name, so organization_id keeps its meaning on every table and in every row-level-security predicate. Everyone a tenant deals with becomes a partners row scoped to that tenant: IMP's Al-Ain and Zaytoun's Al-Ain are two rows even when they are one company, and a partner that later becomes a tenant is linked rather than merged. Contacts belong to a partner and carry as many channel addresses as the person has; the resolver from an address to (tenant, partner, contact) is what gives a conversation its identity and a forward its origin. This is the addressing layer, deliberately kept to three tables and no CRM features; a tenant that runs a CRM makes it the authority for partners and contacts through entity_authority, the same way an ERP is for orders.

3.4 Commercial and freight

One pattern serves every commercial aggregate: a a permanent top-level record, versions that are not edited after creation, and a field identifying the current version.

orders          id PK, organization_id, side (purchase|sale) NOT NULL,
                buyer_partner_id, seller_partner_id, status, current_revision_id
order_revisions id PK, order_id, revision_no, terms, currency, incoterm,
                source_document_id NULL, decided_by_decision_id NULL
                UNIQUE (order_id, revision_no)
order_lines     id PK, organization_id, order_revision_id, product_id NULL,
                qty, uom, unit_price, tax, fulfillment_state

commercial_requests, quotes: same pattern, side NOT NULL.

Freight is its own bounded context (a part of the business with its own model and rules) (lifecycle genuinely differs:
comparison, ranking, equipment, routing):
freight_requests, freight_quotes + freight_quote_charges,
freight_comparisons, freight_allocations, freight_bookings (route legs, equipment)
Principal seams to the rest of the model: order-line allocations,
shipment/container allocations, settlement obligations.

3.5 Logistics and tracking

shipments       id PK, organization_id, mode, direction, shipper/consignee/notify
                partner ids, status
shipment_legs   id PK, shipment_id, leg_no, from/to (port or location),
                vessel_id NULL, voyage NULL, scheduled/actual dates
handling_units  id PK, organization_id, shipment_id,
                kind (container|package|pallet), identifier, seal NULL

allocations                         -- the three-way-match substrate
  id PK, organization_id, order_line_id, shipment_id,
  handling_unit_id NULL, qty, uom

observations                        -- append-only: what a source asserted about ANY object; nothing here is rewritten
  id PK, organization_id, subject_type (shipment|order|order_line|brand|obligation|...), subject_id,
  attribute, value jsonb, modality (expected|actual),
  effective_at NULL, observed_at, received_at,
  source_class (provider|erp|document|message|operator), source_ref,   -- delivery, document or provider poll
  decision_id NULL,                   -- set when the observation came through the review gate
  payload_hash
  UNIQUE (organization_id, payload_hash)   -- the same report arriving twice is one observation

object_state                        -- deterministic reduction per (subject, attribute), versioned
  subject_type, subject_id, attribute, current_value jsonb, modality,
  reducer_version, derived_from uuid[]      -- the observations that produced this value
  PK (subject_type, subject_id, attribute)

Shipment tracking is the first reducer, and its incident rules stay as versioned code invariants with fixtures: origin port write-once; destination changed only when empty or when the five reroute checks all pass; provider telemetry outranks a document's dates; once an actual milestone is confirmed, later inputs cannot change it; BOL refreshes cannot overwrite operator edits. The table is no longer shipment-shaped because shipments are no longer the only object the world keeps reporting on. A supplier's email pushing a lead time out a week is an observation on an order's promised date; a customer's reply is an observation on a line's sign-off; a forwarded file is an observation on a brand's current version. Each (subject_type, attribute) registers a reducer: a pure function from its observations to a current value, with a precedence ladder over source_class and modality and a version number, so a rule change can be replayed and diffed. Message-sourced observations reach this table only through the review gate, so a misread email cannot move state without a policy or a person having seen it.

3.6 Settlement

obligations                         -- one aggregate, both directions
  id PK, organization_id, direction (payable|receivable) NOT NULL,
  debtor_partner_id, creditor_partner_id, currency,
  counterparty_doc_number, posting_state, current_revision_id
  UNIQUE (organization_id, direction, creditor_partner_id,
          counterparty_doc_number, doc_class)          -- the chain identity, enforced once

obligation_revisions
  id PK, obligation_id, revision_no, totals, source_document_id,
  ordering_evidence jsonb,            -- manual pin, revision marker/date, generated ts, doc date, receipt time
  decided_by_decision_id NULL, is_current
  UNIQUE (obligation_id) WHERE is_current             -- one live head; demoted in the same tx

obligation_lines   id PK, organization_id, obligation_revision_id,
                   product_id NULL, order_line_id NULL, qty, price, tax

payments           id PK, organization_id, obligation_id,  -- bound to the obligation;
                   amount, date, method, external_ref     -- revision rows hold no payments
payment_schedules  obligation-bound, same reason

adjustments                         -- credit/debit notes are typed adjustments
  id PK, organization_id, obligation_id,
  adjusted_revision_id,               -- the posted revision it legally adjusts
  kind (credit_note|debit_note), lines, external_doc_ref NULL,
  source_document_id NULL

The current invoices.purpose enum dissolves into its three honest ideas: document class stays on the document, lifecycle is a status, and a credit note is an adjustment with a sign, bound to both the obligation and the revision it adjusts. The materiality gate and the complete ordering ladder are invariants with fixtures; the two current, inconsistent chain-key implementations become one unique constraint

3.7 Evidence bindings, eventing, ERP, policy

evidence_bindings                   -- documents to business objects; many-to-many
  id PK, organization_id, document_id, business_object_id,
  kind (source|evidence), is_primary bool,
  decision_id NULL, superseded_at NULL, superseded_by NULL
  UNIQUE (document_id, business_object_id, kind) WHERE superseded_at IS NULL
  UNIQUE (document_id, object_type, kind) WHERE is_primary AND superseded_at IS NULL
  -- supersession replaces a specific binding; only evidence bindings project,
  -- and the primary flag decides where a single FK projection needs one winner.
  -- source = provenance ("this BOL created this shipment"); it does not project.

domain_events      id PK, organization_id, event_type, subject_type, subject_id, payload, trace, created_at
  -- append-only log written by activities in the same transaction as the fact. A record for audit,
  -- analytics and debugging rather than a transport: side effects run as workflow activities, and a
  -- fact reaches a workflow by SignalWithStart after commit (see section 6, consistency without an outbox).

erp_commands       id PK, organization_id, integration_id, command, payload,
                   status, attempts                    -- durable command ledger
erp_links          integration_id, entity_type, entity_id, remote_id,
                   watermarks, sync_status (incl. parked conflict)

entity_authority                    -- who owns the fact, per tenant and entity type
  organization_id, entity_type (order|customer|product|invoice|...),
  authority (aradus|erp), integration_id -> integrations NULL, since
  PK (organization_id, entity_type)
  -- default aradus when no ERP is connected. Deliveries, dispatches, artifacts, proposals, decisions,
  -- evidence, policies, brand versions and sign-offs are aradus-owned for every tenant and are not listed here.

policies                            -- the config surface; versioned, effective-dated
  id PK, organization_id NULL, counterparty_id NULL, counterparty_role NULL,
  document_type NULL, layout NULL, channel NULL, field_path NULL,
  policy_type (template|check|matching|autonomy|linkage|versioning|review_routing),
  definition jsonb, version, effective daterange, created_by
  -- autonomy definitions carry a MODE as well as thresholds: auto_when_signals_agree | require_approval | ask_below_similarity
  -- every policy_type declares its runtime reader; a test fails if a surfaced
  -- option has no execution path. Incident rules are NOT here: they are
  -- versioned code invariants; policy may route outcomes to review, not weaken them.

audit_events       business_object_id, actor, action, before_ref, after_ref

3.8 What moves, what stays

One rule decides the fate of today's 150 tables: a table moves to the new schema only if the design changes what it means. Everything else stays in public and is reached through it.

4 · Workflows

Every capability is a Temporal workflow written in code, and there are three shapes. Entity workflows live as long as the thing they are about: one per conversation, one per shipment, one per ERP integration. Run workflows do one unit of work and end: a file through the pipeline, a document through extraction, an order through confirmation, a loading list through generation. Schedules replace cron: digests, sweeps, retention, billing close. Model calls are activities and stay out of workflow code. Every value a tenant varies is read from policies at the step that needs it. Nothing an operator can draw; everything an operator can configure.

flowchart TB
    D[deliveries
email, WhatsApp, upload, API] --> CW[ConversationWorkflow
entity: tenant, channel, contact]
    CW -- a question --> CW
    CW -- answers an open question --> ASK[the workflow that asked]
    CW -- a file or attachment --> FR[FileRun
classify and split]
    CW -- text about an object --> DR
    CW -- a forwarded design file --> AR[ArtworkReplacementWorkflow
run per file]
    FR --> DR[DocumentRun
extract, resolve, link, version]
    DR --> GATE{review gate
policy or a person}
    AR --> GATE
    GATE --> MAT[materialize
objects, revisions, observations]
    MAT --> OC[OrderConfirmationWorkflow
run per order]
    MAT --> TR[ShipmentWorkflow
entity per shipment]
    MAT --> ERP[ErpSyncWorkflow
entity per integration]
    P[provider feeds
tracking polls, ERP pulls] --> TR
    P --> ERP
    OP[operator actions
start, decide, generate] --> OC
    OP --> DG[DocGenRun
loading list to documents]
    SCH[schedules
digests, sweeps, retention, billing] --> NT
    OC --> NT[dispatches
notifications, questions, documents out]
    TR --> NT
    ERP --> NT
    DG --> NT
    NT --> OUT[to people and systems
email, WhatsApp, ERP]

Arrows are "wakes" or "writes". Every workflow writes through activities to the same tables; the gate is the same code path for a field, a link, a version, a sign-off or an assertion.

5 · Key flows

The trace in full: one invoice, every write, in order

ACME (partner ptn_9a2) emails invoice INV-4471 for USD 4,200 against shipment shp_113. Every row the pipeline writes, numbered in commit order, showing the values stored in each row. Each row contains IDs linking it to the preceding rows so each write is traceable to the ones before it.

Reads are omitted; the diagram shows writes only, in commit order. Proposals (6, 11–13) and decisions (7, 14, 17) are the only tables written in more than two steps, and both only ever gain rows.

Merged PDF, five documents inside

1. Intake creates one delivery, one artifact, and a file-level run, opened before its documents are split (the CHECK on runs allows exactly this case). No document row yet: nothing is known to be a document.
2. Classification proposes each document's type and page range, one proposal per segment with its confidence. Policy decides them; a low-confidence segmentation pauses for review before anything is sliced.
3. Materialize: per decided segment, create a derived file for each approved page range into renditions, insert a documents row with artifact_part and the parent delivery's provenance, and start a per-document run. A redelivered split re-runs against the same digests and no-ops.
4. Each document proceeds through the happy path independently. A reclassification of the artifact replaces the earlier set while retaining it in history.

Workbooks follow the same flow with sheet names as artifact_part. There is no parent pseudo-document and no fake system actor: The delivery record carries the source information.

Denied segmentation: retry and correction

What happens when the split proposal is wrong. The cost depends entirely on timing: before documents exist, denial is free; after, it is a generation supersession. The gate exists to make the second case rare.

flowchart TD
    P["segmentation proposal<br/>on the file-level run (pass 1)"] --> D{"decision"}
    D -- "reject: wrong, machine<br/>should try again" --> E["escalated re-read:<br/>higher DPI, stronger model"]
    E --> P2["new proposal, pass+1,<br/>same run - prior pass kept"]
    P2 --> D
    D -- "defer" --> W["file run stays<br/>waiting_review"]
    W --> D
    D -- "auto-accept<br/>(policy threshold met)" --> M["materialize: documents born typed<br/>+ per-document runs"]
    D -- "correct: the reviewer's boundaries<br/>become the decided_value" --> M
    M --> OK(["pipeline continues<br/>per document"])
    OK -.-> L["split found wrong LATER,<br/>documents already exist"]
    L --> R["explicit resegment:<br/>new file-level run"]
    R --> NP["new segmentation proposal"]
    NP --> ND{"decision"}
    ND -- "accept / correct" --> G["generation+1 documents,<br/>born typed from the corrected split"]
    G --> SUP["old generation superseded, not deleted:<br/>runs, proposals, decisions, bindings kept;<br/>materialized records reconciled by supersession"]

Reject cannot loop unbounded: each escalated pass produces a new proposal that costs a fresh decision, and correct is the terminal fallback - the reviewer states the truth and documents are created from it. Every wrong guess stays on record as a labeled example.

Duplicate arrival, and re-upload with intent

If the resent file has the same hash, reuse the existing file record: a new deliveries row records the arrival (sender, channel, time) and nothing reprocesses - existing documents keep their original producing delivery and are not retargeted. A manual re-upload whose request metadata states what the operator wants also reuses the stored file: a reprocess request opens new runs on the existing documents; only an explicit reclassify or resegment creates a replacement document generation. Content is stored once in both cases; the difference between "we received it again" and "the operator wants it re-read" is a field on the delivery row rather than a code path.

Revised bill

The new document lands in the same document_family by chain key. The materiality gate compares with the current version: an empty or trivial diff returns the head unchanged, so a re-upload cannot reset an approved bill. A material diff creates a revision proposal; the decision (policy or human, per the tenant's versioning policy) inserts an obligation_revisions row, demotes the old head in the same transaction, and payments stay bound to the obligation. An older document is stored as a non-current revision without changing the current version or its payments.

BOL creates a shipment

1. Extraction and resolution produce proposals for the BL number, parties, ports, containers, and an entity_create proposal for the shipment itself, since no matching shipment exists.
2. On acceptance, commit create,s the shipment, its legs and handling units (trackable containers, packages, or pallets) in one transaction, with the materializations digest guarding redelivery. A container number that fails its check digit materializes the unit with a null identifier and a pending field task; keep the suspected number only as an unapproved proposal; do not use it for tracking.
3. An evidence binding of kind source records that this document created this shipment. It does not populate any derived shipment field; a duplicate BOL arriving later gets an evidence binding only.
4. Tracking initialization appends the first observations; the reducer computes initial state; from here provider polling and the BOL disagreeing about dates is the reducer's problem, under its versioned precedence rules.

Ambiguous shipment link

Two candidate shipments for one certificate: The proposed match stores both candidate shipments and the evidence used to compare them including the document's own dates, so a container reused eight months apart resolves without a person. A genuine tie leaves the link proposal pending; the human review task shows both candidates with evidence; the decision creates the evidence binding, and derive updates the projections. A later manual re-link supersedes that specific binding and re-derives. No scheduled retry calls and charges the model again while review is pending while the task sits open.

A correction that teaches

1. An operator corrects the seller on a pending entity-match proposal: the decision row records decided_value beside the untouched proposal, the actor, and the task.
2. The same transaction writes a scoped alias ("this counterparty's 'ACME TRDG LLC.' means party X") with created_from_decision_id, subject to the an alias may identify only one record; a conflicting alias opens a review task; a collision with an existing alias opens an ambiguity task instead of silently overwriting.
3. The proposal-decision pair becomes a labeled example: it joins the calibration set for that field and counterparty, and the few-shot retrieval pool for the next document from that sender, with evaluation cases kept separate.
4. The next invoice from this supplier matches the seller exactly from the saved alias before calling a model, before any model runs.

Reprocess and change type

Reprocess opens a new run; the old run and its proposals stay, marked superseded. New proposals are decided (prior human decisions auto-apply where subject and value match; the rest re-enter review), commit re-materializes through the same digests, updating rows in place where the target exists, and evidence bindings are superseded and recreated rather than deleted. A change of type does not update document_type in place: it is a human-decided classification proposal that creates a replacement document generation, supersedes the old one, and reconciles its evidence bindings and materialized records. Nothing is hard-deleted, so a reprocess cannot destroy corrections, and the a safeguard requiring approval when changing the type would leave a shipment without its creating document is a policy rule: a type change that would orphan a shipment routes to an approval task instead of being forbidden in code.

ERP two-sided conflict

1. The inbound pull detects that both Aradus and the ERP changed the same record since the last sync. The erp_links row is marked as a conflict; the conflict remains paused until a person resolves it; routine synchronization must not resume it, since the park's own bookkeeping must not un-park it on the next sweep.
2. An adjudication task of kind erp_conflict opens with both versions and the field diff. No writes flow in either direction while it is open.
3. The operator picks theirs, ours, or a merge. The decision releases the conflict, and the outcome becomes an erp_commands row (ours) or an inbound patch (theirs), each idempotent under the customer's compatibility contract.

Text-only message about an existing object

1. A carrier emails "MSC Adriana will discharge at Khalifa Port, ETA 27 March." The delivery has one body artifact; classification proposes correspondence and the document is born typed.
2. The run produces a refers_to proposal and two assertion proposals: destination port = Khalifa (actual) and ETA = 27 March (expected), each with a text-span source_ref.
3. Policy for this tenant trusts carrier senders about routing, so the port assertion auto-accepts and materializes an observation. The ETA assertion auto-accepts as expected, and a notification fires because the slip exceeds the tenant's two-day rule. Both materializations key on the shipment instance, so a second email about a second container updates a second shipment without collision.
4. When the provider feed later reports the actual berth, that observation outranks the expected one and the state moves. The email's observation stays on record as what the carrier said.

A person messages first

A contact sends "cancel line 3 of yesterday's order" with nothing pending. Intake does SignalWithStart on conversation-{tenant}-{channel}-{contact}. The conversation workflow's interpret activity classifies the message as an instruction about an object, resolves "yesterday's order" against the contact's recent deliveries, and produces an assertion proposal (line 3 status = cancelled) that goes through the same gate as any other. Had the message answered an open question, the conversation would have signalled the workflow that asked. Had it been a question, an activity would have read current state and drafted a reply as a dispatch. Every turn is a delivery or a dispatch row; the workflow holds only the open questions and pointers to the last few turns, and continues as new after a bounded number of turns.

6 · Deep dives

Confidence and calibration

The gate does not read a model's self-assessment. Inputs to the calibrated score: cross-pass agreement (pass 2 uses a different model or rendering, since two correlated passes are weak evidence), deterministic validator results, resolver features, and the counterparty's track record. The calibrator maps raw scores to observed correction rates (adjusts model scores using how often similarly scored values were actually corrected) per document type, field and layout. Its training data has three sources by design: gold cases (manually verified examples with known correct answers), production corrections, and random audits of auto-accepted work, because a dataset containing only corrections is biased toward cases that were already sent to reviewers. Hierarchical priors cover cold starts; retrieval examples are kept out of evaluation sets.

Policy versus invariant

Configuration is applied from platform defaults down to field-specific rules (platform → tenant → counterparty and role → document type → layout → channel → field). It holds statements of business fact: templates with field-to-column maps, checks with tolerances and severities, matching strategy order, linkage conventions, materiality rules, autonomy grants. Use centrally managed thresholds based on observed error rates; customer-specific thresholds would overfit small datasets. Incident-derived correctness rules are not policy at all: the ordering ladder, tracking precedence and ERP idempotency are versioned code with fixtures, and policy can only choose whether an outcome routes to review.

Idempotency where money moves

Three ledgers make repeated delivery harmless (three durable operation records prevent retries from repeating billable or financial effects). model_invocations: a call is reserved under a deterministic ID before dispatch, so a crash after the provider answered is reconciled, not re-billed, and billing reads this table rather than queue traffic. materializations: A repeated commit matches the existing input hash and makes no change. erp_commands: every push is a durable command with attempts, and the customer-facing semantics (external ID schemes, for products first created in the ERP, treat an exact SKU match as the same product, case-insensitive number matching within tenant and direction, draft-only posting, already-posted no-ops, line replacement, conflicts that stay paused until a person resolves them) are a per-customer compatibility contract, changed only after certification against that customer's system.

Messages as input: six changes

Documents arrive with their own identifiers and a fixed shape; messages do not. Six changes make a message a first-class input rather than an attachment carrier. Bodies are artifacts and deliveries carry threading (3.1). Linkage is its own proposal type, refers_to, with its own threshold, because for a message the hard question is which object it is about rather than what it says (3.2). Evidence anchors include a text span, so a review shows the quoted words rather than a page (3.2). Statements about existing objects are assertion proposals that materialize observations, and the reducer covers every subject type rather than only shipments (3.5). Materialization is keyed by target instance, so one message can update several objects of one type (3.2). Extraction schemas for correspondence are assertion-shaped: a variable-length list of (subject, attribute, value, modality, effective time) rather than fixed fields, declared as a document type whose schema allows a list. The delivery is the unit of context for all six: one message, one body, N attachments, read together.

Who owns which fact

Two kinds of record are easy to confuse. The ERP records business facts: an order, an invoice, a customer. Aradus records how a fact came to be: the message it arrived in, what the machine proposed, what a person decided, the evidence, and the rule that came out of it. The second kind is ours for every tenant and is the memory that makes the system better per customer; no ERP holds it. The first kind is owned per tenant and per entity type through entity_authority. When an ERP is connected it originates and corrects the fact, our row mirrors it with the remote id, and an accepted proposal becomes an erp_commands push rather than a local write. When no ERP is connected, we are the record, edits happen here, and an ERP added later is a downstream target. Same tables in both modes; the flag decides who may originate a fact and which way sync runs. The consequence is accepted: for a tenant with no ERP, being the record of their orders means being their ERP.

Conversations are workflows

Headless and conversational are the same step at two points on the confidence scale. A clean spreadsheet order and a photo of a handwritten one both produce a sales-order proposal; one clears the tenant's threshold and materializes, the other becomes a question. The question is a dispatch to the person on their channel, the reply is a delivery, and the reply is read as a decision on the open proposal. So a chat is a workflow: state is the set of open questions and pointers to recent turns, an inbound message is a signal, a model call is an activity (one to phrase the question, one to read the answer), an outbound message is a dispatch. There is no separate agent; the workflow is the agent, and headless is the same code with the question step skipped. Messages a person starts land in a per-contact entity workflow keyed conversation-{tenant}-{channel}-{contact}, created by SignalWithStart, which routes each message as an answer to an open question, an instruction about an object, or a question to be answered from state. Its id gives the tenant discrimination the current WhatsApp routing lacks. Automated feeds stay on the observation path and are kept out of conversations, because a workflow processes its signals one at a time.

Runtime: self-hosted Temporal, control flow in code

Temporal is self-hosted on the existing Kubernetes cluster for data residency, with the CNPG Postgres operator as persistence. It is a durable execution runtime for code and offers no visual authoring layer. The current system's Engine B bundles two things: durable execution, done by hand with rows, a poller and a seven-day stale sweep, and a graph builder operators can edit. The greenfield keeps neither. Control flow is code, as Temporal workflows, versioned like code: match, send, wait, remind, convert is the same shape for every tenant. Everything a tenant varies is policy data read by the workflow at each step: who to send to, how often to remind, which threshold auto-accepts, whether a marked-up screenshot counts, which mode a replacement runs in (auto_when_signals_agree, require_approval, ask_below_similarity). The operator-facing surface is a rulebook rather than a canvas. Model calls live only in activities; workflows carry ids rather than customer data; long chases continue as new; and worker versioning is decided before the first workflow runs for weeks, since a deploy that changes a running workflow's logic without it fails on replay.

Consistency without an outbox

A Postgres commit and a Temporal call cannot be one atomic operation, and the design uses Temporal's own answer rather than a relay. For actions a person takes (Start, a decision, remind now) the API updates or signals the workflow and the workflow's activity writes the row: the workflow is the source of truth while it is open, and Postgres holds the projection the UI reads. Temporal's founder gives this as the recommended pattern. For arrivals from outside the handler writes the delivery idempotently on the provider's message id, commits, then does SignalWithStart on the deterministic workflow id with reuse policy reject-duplicate. The provider retries until it receives a 2xx, so a crash between commit and signal is healed by the retry hitting the same idempotent write and the same idempotent start. Temporal's guidance for this case is the same: an idempotent transaction plus an idempotent start needs no outbox. What is given up: a mutation depends on Temporal being reachable, any writer that bypasses the application must start workflows explicitly, and deduplication moves from one relay into every activity. What is gained: no relay, no broker on this path, and replay by resetting a workflow. domain_events stays as an append-only record written by activities, for audit and analytics, and is not read by anything to trigger work.

7 · Migration plan

The company is under six months old and the current system has two paying tenants, so this is a plan for moving a small live system in pieces rather than cutting over a large one. Decisions that shape it: the core moves and the edges stay; capabilities move in dependency order and are switched on per tenant; the graph builder is frozen from today; history moves forward from each cutover; the greenfield gets a new schema on the existing Postgres instance.

End state

Everything that reads, decides, remembers or waits runs on the spine: intake as deliveries, the document pipeline, invoice versioning, tracking reduction, every workflow as Temporal code, memory as proposals, decisions, observations and policies. Everything that is an adapter to the outside is kept and re-pointed: the Next app with its auth and RBAC, the provider clients (SeaRates and the other tracking vendors, Twilio, Resend, Graph), the ERP connectors, the doc-gen renderer, Slack. Old Node and Go business logic is deleted when its last reader has moved; adapters move into the new codebase afterwards as file moves. Engine B is frozen now: no new graphs or node types, existing graphs run until each is rewritten as a workflow.

Order

Steps 1 and 2 are the seams every later step shares, which is why they go first and why they move behaviour without improving it: every tenant's traffic passes through them. From step 3 on, each step is a set of readers moving to tables that already hold production data.

History

Each capability starts on the spine with current state: live documents, current brand versions, open orders, and one baseline observation per shipment labelled as a legacy baseline. Older rows stay in public, readable through a legacy view in the Command Center and reports, and are left where they are. Provenance stays honest because nothing is invented, and the old tables are dropped when retention ends rather than after a port.

Database

One Postgres instance. The greenfield owns a new schema with its own migrator and row-level security on every table; nothing in it references public by foreign key. The Next app and the Go API reach both schemas over one connection, so a read model that shows an old order beside its new sign-offs is one query during the transition. The two-migrator drift that produced the ARD-330 incident cannot recur, because each schema has exactly one migrator. Section 3.8 lists every current table and where it ends up.

What parity means

Parity is user-observable capability, listed before anything is built: channels, duplicate rules, file types, document types, fields, resolutions, mutations, links, reviews, statuses, notifications, metering, tracking rules, ERP operations, retry and give-up paths, permissions, audit. Incident rules become characterization tests carrying their original examples. Table shapes and the current system's two inconsistent ways of versioning a bill are implementation detail, not parity. The eval corpus and CI gates exist from step 3 onward; each capability adds its gold cases before its model calls ship.

Produced 4 Sep 2026 against branch next; revised 9 Sep 2026 with messages as input, authority per tenant and entity, conversations as workflows, the self-hosted Temporal runtime, consistency without an outbox, and the migration plan in section 7. Companion pages: Order Confirmation Design and Aradus Field Guide. Earlier companions: Document Pipeline ERD and Pipeline Write Paths.