Greenfield Pipeline Design

1 · Problem and requirements

Aradus converts untrusted, counterparty-authored documents into shipment, payment, compliance, tracking and accounting records, and those records drive the customer's operations and their accounting systems. A perfectly extracted field can still be wrong in every way that matters: attached to the wrong shipment, matched to the wrong legal entity, duplicated into a second bill, treated as older than an outdated version, posted twice to an ERP (enterprise resource planning) system. So the product is correctly matching records, preserving their sources, resolving conflicts, selecting the current version, and writing only approved data. Extraction is the easy half.

Four kinds of record must stay separate. A delivery is how something arrived and from whom. An artifact is the file itself - the PDF or spreadsheet as received, stored once and not modified afterwards - possibly shared by many deliveries (the same attachment sent twice is one artifact, two deliveries). A document is a distinct business document within the file, created once classification has decided it exists; one artifact may hold five documents, and one document may arrive repeatedly or revised. A run is one interpretation using recorded, fixed versions of the code, model, prompt, schema, and policy (configured rules deciding what may be automated and what requires review). Most of the current system's complexity traces to collapsing these four into one table.

Functional requirements

1. Ingest via upload, email, chat, API (receive files through upload, email, chat, or API); merged PDFs split into documents; process each workbook sheet separately; store one copy of byte-for-byte identical files while recording every arrival with record the sender, channel, and time for every arrival.
2. Classify, extract, normalize (convert extracted values into standard formats) and validate roughly 25 document types; about nine of them create or update business records (invoice, bill of lading (BOL), air waybill (AWB), order, freight quote, packing list, two certificates, courier receipt); the rest are evidence.
3. Match extracted names and codes to supplier, product, port, carrier, and vessel records with alternate names that apply only to a particular customer or supplier; match, create and correct are three separate decisions.
4. Link documents to shipments and orders; rank candidates using identifiers, sender, dates, parties and route; send genuinely ambiguous matches for human review.
5. Version supplier bills: Check whether the changes are significant enough to create a new bill version, fixed tie-break rules for deciding which revision is current, payments remain attached to the bill's current version; revisions for every other document type too.
6. Route uncertain values to human review; every correction becomes training and evaluation data; approvals and corrections both become evaluation and training examples.
7. Synchronize records with Odoo and Business Central. (Odoo, Business Central) and tracking without changing the accounting behavior customers already rely on.
8. Reprocess and change type without destroying history; per-tenant (per customer organization) document type catalog; per-counterparty templates and conventions set up through configuration rather than code.
9. Record the usage and cost of every model call for billing, including failed calls.

Non-functional requirements

1. Multi-tenant with hard isolation (one shared system serving multiple customer organizations while strictly separating their data); a bug in one tenant's query cannot read another's rows.
2. At-least-once delivery (each message is delivered one or more times, so duplicates are possible) everywhere; every effect idempotent (safe to retry: repeating the same operation does not create a second effect); a billable model call is not repeated on redelivery.
3. Every accepted value traceable to its evidence, machine proposal, policy and decision, with the actor recorded.
4. Accuracy measurable per stage, per tenant, per document type; changes in accuracy can be traced to a model, prompt, or dataset change.
5. Automatic approval can be enabled or disabled by field, counterparty, and document type; gated on calibrated (adjusted so a stated confidence matches the observed error rate) confidence, not model self-assessment.
6. Evidence retained per tenant policy with legal hold (a requirement to preserve evidence because of litigation, investigation, or regulation); a customs dispute runs years.
7. Onboarding a new counterparty format costs configuration, not engineering.

Scale honesty

This is not a throughput problem (a limit on how many documents the system can process per unit of time). Current production creates on the order of one shipment a day; even at 100x, Postgres on one primary handles every table here without partitioning. The scarce resources are model spend, human review minutes, and correctness under retries. The design optimizes for those three, and for the growing value of the corrections dataset, not for request throughput.

2 · Architecture

Five components, and the reason each earns its place:

• One modular Go codebase deployed as API and worker pools (groups of background processes that execute queued work). No microservices: the transactional boundaries (groups of changes that must either all commit or all fail) around documents, obligation (a payable or receivable amount owed)s, shipments and reviews matter more than deployment independence at this scale.
• Managed Temporal manages workflow scheduling, retries, timers, and recovery: leases (time-limited ownership of a task by one worker), retries, timers, heartbeats, split fan-out and fan-in, review waits, cancellation, workflows continue safely across deployments. Postgres stays the only business truth; a uI or API reads run status from Postgres projection (a query-friendly field or view derived from the authoritative records)s, not from Temporal.
• A Python media and model gateway, deliberately shrunk: PDF and image rasterization (PyMuPDF has no Go equal), provider adapters and their quirks, and the the process for testing and improving extraction prompts and examples (few-shot assembly (selecting a small set of worked examples to include in a model prompt), future future automated prompt optimizers). It is stateless (the service keeps no durable business data between requests), has no database credentials, and schemas are defined once in JSON Schema and used to generate matching Go and Python types, so the the Go and Python schema definitions becoming inconsistent the current system suffers cannot recur. If the PDF tooling gap ever closes, this component can move into the Go application; nothing else depends on its language.
• PostgreSQL, one primary: Authoritative stored data, row-level security provides a second enforcement layer between customers' data, transactional outbox (a database table of events written in the same transaction as the business change, then delivered reliably) with one delivery row per subscriber.
• S3: the original files (written once, then read-only) plus derived files such as page images and crops, which are regenerable copies rather than authoritative source data.

3 · Data model

erDiagram
    deliveries }o--|| artifacts : "artifact_id - many arrivals, one stored file"
    artifacts ||--o{ renditions : "derived page images and crops"
    artifacts ||--o{ documents : "artifact_id"
    deliveries ||--o{ documents : "delivery_id - the producing arrival, same file enforced"
    pipeline_proposals |o--o{ documents : "created_from_proposal_id UNIQUE - born typed"
    document_families ||--o{ document_family_members : "family_id"
    documents ||--o| document_family_members : "document_id UNIQUE"
    artifacts |o--o{ pipeline_runs : "file-level runs (classification)"
    documents |o--o{ pipeline_runs : "per-document runs - CHECK exactly one of"
    pipeline_runs ||--o{ pipeline_proposals : "run_id"
    pipeline_runs ||--o{ model_invocations : "reserved before dispatch - billing truth"
    pipeline_proposals ||--o{ proposal_decisions : "append-only verdicts - latest wins"
    proposal_decisions |o--o{ adjudication_tasks : "task_id when a person decided"
    proposal_decisions |o--o{ entity_aliases : "a correction writes a scoped alias"
    proposal_decisions |o--o{ evidence_bindings : "an accepted link creates the edge"
    documents ||--o{ evidence_bindings : "document_id"
    evidence_bindings }o--|| business_objects : "typed target - an enforced FK"
    pipeline_runs ||--o{ materializations : "accepted set digest - idempotent persist"
    materializations }o--|| business_objects : "the domain record it created"
    business_objects ||..o{ domain_aggregates : "orders freight shipments obligations..."
    organizations ||--o{ pipeline_policies : "the config cascade"

    deliveries {
        varchar id PK
        varchar organization_id FK
        intake_channel channel
        varchar contact_id FK "sender identity"
        varchar artifact_id FK "NOT NULL"
        varchar idempotency_key UK
    }
    artifacts {
        varchar id PK
        varchar organization_id FK
        text content_hash "UNIQUE per org"
        text storage_tier "cost"
        timestamptz retain_until "obligation"
    }
    documents {
        varchar id PK
        varchar artifact_id FK "NOT NULL"
        varchar delivery_id FK
        varchar created_from_proposal_id FK "UNIQUE - retry-safe identity"
        integer generation "resegment supersedes the set"
        text artifact_part "pages or sheet"
        varchar document_type "NOT NULL - born typed"
    }
    pipeline_runs {
        varchar id PK
        varchar document_id FK "nullable"
        varchar artifact_id FK "nullable - CHECK one of"
        run_status status "running waiting_review completed failed superseded"
        varchar policy_snapshot_id
    }
    pipeline_proposals {
        varchar id PK
        varchar run_id FK
        proposal_type proposal_type "classification field entity_match entity_create link revision"
        varchar subject
        smallint pass "UNIQUE run+type+subject+pass"
        jsonb proposed
        jsonb source_ref "page bbox - what review highlights"
        numeric confidence "calibrated"
        jsonb produced_by "model prompt schema policy versions"
    }
    proposal_decisions {
        varchar id PK
        varchar proposal_id FK
        decision decision "accept correct reject defer"
        jsonb decided_value "corrections land here"
        varchar actor_type "policy user import process"
        varchar supersedes_decision_id FK "re-decisions append"
    }
    materializations {
        varchar id PK
        varchar document_id FK
        varchar business_object_id FK
        text proposal_set_digest "UNIQUE(document, target_type)"
    }
    evidence_bindings {
        varchar document_id FK
        varchar business_object_id FK
        binding_kind kind "source or evidence - only evidence projects"
        boolean is_primary
        timestamptz superseded_at "append-only"
    }
    business_objects {
        varchar id PK
        varchar organization_id FK
        varchar object_type "identity spine - no attributes"
    }

Overview of the pipeline core and its seams; the DDL blocks below are authoritative for columns and constraints, and the domain aggregates (orders, freight, shipments, obligations) are drawn in sections 3.4–3.7.

Conventions: every tenant-owned table carries organization_id with composite foreign keys (foreign keys containing both the organization ID and record ID) and RLS (postgreSQL row-level security (RLS), which restricts each query to the customer's rows); global reference data (countries, currencies, ports, carriers, vessels) does not. created_at/updated_at everywhere, omitted below. Types are abbreviated. Critical database constraints that enforce the design are spelled out because they are the design.

3.1 Intake and evidence

deliveries                          -- every arrival, repeat provenance kept
  id PK, organization_id, channel (uploaded|email|chat|api|generated),
  actor_type (user|system|counterparty), actor_id NULL,
  contact_id -> contacts NULL,        -- the sender: strongest identity signal, captured here
  source_message_ref NULL, requested_context jsonb NULL,  -- shipment hint, forced type, reprocess intent
  idempotency_key, artifact_id -> artifacts, received_at
  UNIQUE (organization_id, channel, idempotency_key)

artifacts                           -- the received file, stored once, addressed by its hash, read-only after write
  id PK, organization_id, content_hash, byte_size, mime_detected,
  storage_key, storage_tier (hot|cold|archive),        -- cost, ops-owned
  retain_until NULL, legal_hold bool                   -- obligation, tenant-owned
  UNIQUE (organization_id, content_hash)

renditions                          -- derived, reproducible; cache not truth
  id PK, artifact_id, kind (page_png|text_layer|region_crop|sheet_file),
  locator, dpi NULL, storage_key

documents                           -- the DECIDED semantic unit; zero self-references
  id PK, organization_id, artifact_id, delivery_id,    -- which arrival produced it
  created_from_proposal_id UNIQUE,    -- the decided classification proposal: a retry cannot create a duplicate document
  generation,                         -- reclassification supersedes the prior set; one current generation per artifact
  artifact_part NULL,                 -- page range or sheet name; NULL = whole artifact
  document_type NOT NULL,             -- born typed: a document exists only once classification has decided
  extraction_variant NULL, current_run_id NULL, deleted_at NULL
  FK (organization_id, document_type) -> document_types
  FK (artifact_id, delivery_id) -> deliveries (artifact_id, id)   -- the producing delivery must be of the same file

document_families                   -- version chains with enforced identity
  id PK, organization_id, document_type, chain_key
  UNIQUE (organization_id, document_type, chain_key)

document_family_members
  family_id, document_id UNIQUE, revision_no, is_current,
  decided_by_decision_id NULL
  UNIQUE (family_id, revision_no)
  UNIQUE (family_id) WHERE is_current   -- one live head; demotion in the same tx

What this buys, versus the current model: documents loses all four self-referencing foreign keys (columns that point to another row in the same table). Documents split from a larger file are rows with an artifact_part; a merged PDF is an artifact, and no placeholder document row for the unsplit file exists for it. Duplicate handling is a constraint plus a delivery row instead of a unused column and two inconsistent code paths. Retention and tiering (choosing the file's storage class based on cost and retention needs) are separate fields with separate owners.

3.2 Processing: runs, proposals, decisions

runs                                -- product-facing projection; Temporal owns execution
  id PK, organization_id, document_id NULL, artifact_id NULL,
  purpose (pipeline|reprocess|shadow),
  status (running|waiting_review|completed|failed|superseded),
  code_version, policy_snapshot_id, temporal_workflow_id
  CHECK (num_nonnulls(document_id, artifact_id) = 1)   -- classify/split runs are artifact-level
  UNIQUE (document_id) WHERE status IN ('running','waiting_review')
  UNIQUE (artifact_id) WHERE status IN ('running','waiting_review')
                       AND document_id IS NULL   -- one active file-level run per file

stage_executions                    -- append-only attempts
  id PK, run_id, stage, attempt, input_digest, status, output_ref
  UNIQUE (run_id, stage, input_digest)  -- a retry returns the recorded result instead of re-paying

pipeline_proposals                     -- IMMUTABLE machine observations
  id PK, organization_id, run_id, document_id NULL, stage,
  proposal_type (classification|segment|field|entity_match|entity_create|link|revision|observation),
  subject, pass smallint,             -- pass 2 = targeted high-fidelity re-read
  proposed jsonb, normalized jsonb,
  source_ref jsonb,                   -- page, bbox / sheet, cell: what the review UI highlights
  features jsonb,                     -- candidate set, agreement, validator results
  confidence numeric NULL,            -- calibrated, not self-reported
  produced_by jsonb                   -- model, prompt, schema, template, policy versions
  UNIQUE (run_id, proposal_type, subject, pass)

proposal_decisions                  -- APPEND-ONLY; latest per proposal wins
  id PK, organization_id, proposal_id,
  decision (accept|correct|reject|defer), decided_value jsonb NULL,
  actor_type (policy|user|import|process), actor_id, rationale NULL,
  task_id NULL, supersedes_decision_id NULL, decided_at

adjudication_tasks                  -- one envelope, typed handlers per kind
  id PK, organization_id,
  kind (field|entity|association|version|erp_conflict|approval),
  business_object_id NULL, document_id NULL,
  payload jsonb,                      -- schema per kind; handlers own transitions + authz
  status (open|resolved|dismissed), assignee NULL

materializations                    -- idempotent persist record
  id PK, organization_id, run_id, document_id, business_object_id,
  target_type, proposal_set_digest
  UNIQUE (document_id, target_type)   -- replaces the "already exists" substring match

model_invocations                   -- the billing source of truth
  id PK (deterministic, caller-supplied), organization_id, run_id NULL, purpose,
  model, prompt_version, schema_version, request_fingerprint,
  status (reserved|completed|failed|ambiguous|reconciled),
  provider_generation_id NULL, tokens_in, tokens_out, tokens_cached,
  provider_cost NULL, latency_ms, reserved_at, completed_at NULL

This trio is the core of the design. Proposals do not change: they are what the machine said, and they are the training labels (known expected values used to train or evaluate models). Decisions only add new rows without changing earlier ones: A later decision replacing an earlier one, a second reviewer, a reprocess each add a row and replace while retaining the earlier version, so history survives. Domain rows are written only from accepted proposal (a stored value or relationship proposed by the system, before approval)s, recorded in materializations with an input digest (a hash that uniquely represents the accepted input set), so a redelivered message matches the existing record and makes no change. Corrections carry both the untouched proposal and the decided value, which is what makes each one a labeled example (an input paired with the accepted or corrected expected value). Accuracy per stage is a GROUP BY over these tables; the eval corpus grows as a side effect of operating the product.

3.3 Identity

business_objects                    -- identity spine: id, tenant, type. No attributes.
  id PK, organization_id, object_type

parties        id PK, organization_id, legal_name, country_code, address, ...
party_roles    party_id, role (customer|supplier|forwarder|other), default_terms NULL
contacts       id PK, organization_id, party_id, email, phone, name
products       id PK, organization_id, sku NULL, name, uom, hs_code NULL, ...

identifiers                         -- deterministic tier: matched before anything fuzzy
  id PK, organization_id NULL,        -- NULL = identifier of a global entity
  entity_type, entity_id, id_type (tax_id|sku|unlocode|scac|imo|iata_prefix),
  value
  UNIQUE (COALESCE(organization_id,''), id_type, value)

aliases                             -- learned vocabulary, scoped, provenance-bearing
  id PK, organization_id, entity_type, entity_id,
  counterparty_id -> parties NULL,    -- scoped wherever the counterparty is known
  alias_text, normalized_text,
  source (correction|operator|import|promoted),
  created_from_decision_id NULL, effective daterange NULL
  UNIQUE (organization_id, entity_type, normalized_text, COALESCE(counterparty_id,''))
  -- one meaning per scoped string; a second entity behind the same string is a review task

global reference (platform-owned, no tenant key):
  countries, currencies, ports, carriers, vessels

Resolution order: identifiers, then scoped aliases, then exact matches on standardized names, then finding possible matching records with a calibrated ranker over retained features (a ranking model whose confidence is checked against observed outcomes, using saved matching signals such as identifiers, dates, and parties). A model chooses only between closely matched candidates, and it gets the document's dates. Match, create and correct are three decision kinds; auto-creating master data (authoritative supplier, product, port, carrier, and vessel records) is a records are created only when configuration permits it; extraction alone cannot create them; extraction has no create path. Product resolution reads the alias table, which fixes the single most expensive dead end in the current system: aliases written on every review and read by nothing.

3.4 Commercial and freight

One pattern serves every commercial aggregate (a business object and the records that must change with it, such as an order with its versions and lines): a a permanent top-level record, versions that are not edited after creation, and a field identifying the current version.

orders          id PK, organization_id, side (purchase|sale) NOT NULL,
                buyer_party_id, seller_party_id, status, current_revision_id
order_revisions id PK, order_id, revision_no, terms, currency, incoterm,
                source_document_id NULL, decided_by_decision_id NULL
                UNIQUE (order_id, revision_no)
order_lines     id PK, organization_id, order_revision_id, product_id NULL,
                qty, uom, unit_price, tax, fulfillment_state

commercial_requests, quotes: same pattern, side NOT NULL.

Freight is its own bounded context (a part of the business with its own model and rules) (lifecycle genuinely differs:
comparison, ranking, equipment, routing):
freight_requests, freight_quotes + freight_quote_charges,
freight_comparisons, freight_allocations, freight_bookings (route legs, equipment)
Principal seams to the rest of the model: order-line allocations,
shipment/container allocations, settlement obligations.

3.5 Logistics and tracking

shipments       id PK, organization_id, mode, direction, shipper/consignee/notify
                party ids, status
shipment_legs   id PK, shipment_id, leg_no, from/to (port or location),
                vessel_id NULL, voyage NULL, scheduled/actual dates
handling_units  id PK, organization_id, shipment_id,
                kind (container|package|pallet), identifier, seal NULL

allocations                         -- the three-way-match substrate
  id PK, organization_id, order_line_id, shipment_id,
  handling_unit_id NULL, qty, uom

tracking_observations               -- append-only; nothing here is ever rewritten
  id PK, organization_id, shipment_id,
  source (provider|document|operator), observed_at, received_at,
  identifier_type, payload jsonb, payload_hash, provenance_ref

tracking_state                      -- deterministic reduction, versioned
  shipment_id PK, reducer_version, current fields + milestones

The reducer (a deterministic function that combines tracking observations into the shipment's current state) encodes the incident rules as versioned invariants with fixtures (rules that cannot be weakened, identified by version and covered by fixed test examples): origin port write-once; destination changed only when empty or when the five reroute checks all pass; provider telemetry (tracking updates received directly from the carrier or tracking provider) outranks document ETA; once an actual milestone is confirmed, later inputs cannot change it; BOL refreshes cannot overwrite operator edits. Reprocessing a document cannot damage tracking because observations only append and the reducer is a pure function (the same observations and rule version produce the same state each time, with no outside side effects).

3.6 Settlement

obligations                         -- one aggregate, both directions
  id PK, organization_id, direction (payable|receivable) NOT NULL,
  debtor_party_id, creditor_party_id, currency,
  counterparty_doc_number, posting_state, current_revision_id
  UNIQUE (organization_id, direction, creditor_party_id,
          counterparty_doc_number, doc_class)          -- the chain identity, enforced once

obligation_revisions
  id PK, obligation_id, revision_no, totals, source_document_id,
  ordering_evidence jsonb,            -- manual pin, revision marker/date, generated ts, doc date, receipt time
  decided_by_decision_id NULL, is_current
  UNIQUE (obligation_id) WHERE is_current             -- one live head; demoted in the same tx

obligation_lines   id PK, organization_id, obligation_revision_id,
                   product_id NULL, order_line_id NULL, qty, price, tax

payments           id PK, organization_id, obligation_id,  -- bound to the obligation;
                   amount, date, method, external_ref     -- revision rows hold no payments
payment_schedules  obligation-bound, same reason

adjustments                         -- credit/debit notes are typed adjustments
  id PK, organization_id, obligation_id,
  adjusted_revision_id,               -- the posted revision it legally adjusts
  kind (credit_note|debit_note), lines, external_doc_ref NULL,
  source_document_id NULL

The current invoices.purpose enum dissolves into its three honest ideas: document class stays on the document (a customs invoice is evidence), lifecycle is a status (a proforma is a a preliminary commercial record that does not yet create a payable or receivable linked to the eventual obligation), and a credit note is an adjustment with a sign, bound to both the obligation and the revision it adjusts. The materiality gate and the complete ordering ladder are invariants with fixtures; the two current, inconsistent chain-key implementations become one unique constraint

3.7 Evidence bindings, eventing, ERP, policy

evidence_bindings                   -- documents to business objects; many-to-many
  id PK, organization_id, document_id, business_object_id,
  kind (source|evidence), is_primary bool,
  decision_id NULL, superseded_at NULL, superseded_by NULL
  UNIQUE (document_id, business_object_id, kind) WHERE superseded_at IS NULL
  UNIQUE (document_id, object_type, kind) WHERE is_primary AND superseded_at IS NULL
  -- supersession replaces a specific binding; only evidence bindings project,
  -- and the primary flag decides where a single FK projection needs one winner.
  -- source = provenance ("this BOL created this shipment"); it does not project.

outbox             id PK, topic, event_type, payload, trace, created_at
outbox_deliveries   event_id, subscriber, status, attempts, next_attempt_at
  -- one row per subscriber: a lost intent is detectable and retryable per consumer

erp_commands       id PK, organization_id, integration_id, command, payload,
                   status, attempts                    -- durable command ledger
erp_links          integration_id, entity_type, entity_id, remote_id,
                   watermarks, sync_status (incl. parked conflict)

policies                            -- the config surface; versioned, effective-dated
  id PK, organization_id NULL, counterparty_id NULL, counterparty_role NULL,
  document_type NULL, layout NULL, channel NULL, field_path NULL,
  policy_type (template|check|matching|autonomy|linkage|versioning|review_routing),
  definition jsonb, version, effective daterange, created_by
  -- every policy_type declares its runtime reader; a test fails if a surfaced
  -- option has no execution path. Incident rules are NOT here: they are
  -- versioned code invariants; policy may route outcomes to review, not weaken them.

audit_events       business_object_id, actor, action, before_ref, after_ref

4 · Key flows

The trace in full: one invoice, every write, in order

ACME (party pty_9a2) emails invoice INV-4471 for USD 4,200 against shipment shp_113. Every row the pipeline writes, numbered in commit order, showing the values stored in each row. Each row contains IDs linking it to the preceding rows so each write is traceable to the ones before it.

Reads are omitted; the diagram shows writes only, in commit order. Proposals (6, 11–13) and decisions (7, 14, 17) are the only tables written in more than two steps, and both only ever gain rows.

Emailed supplier invoice, happy path

1. Email arrives: upsert artifacts by hash, insert deliveries with the sender's contact_id, open a file-level run, write artifact.received (carrying the delivery, artifact and run IDs) to the outbox. One transaction; no document row yet. A Temporal workflow starts.
2. Classify: model call through the gateway (ledger (a durable record of attempted and completed operations) row reserved, then completed); classification proposals inserted; policy auto-accepts above its threshold. Below it the run pauses for review, as waiting_review and the pending proposal appears in the review queue; the decision's outbox event resumes the workflow.
3. Extract, validate, resolve, link: proposals per field with coordinates, observation proposals from deterministic checks, entity-match proposals (the sender's contact resolves the counterparty before any fuzzy matching (approximate name matching rather than exact identifier matching)), ranked link proposals carrying the document's dates. Fields whose checks disagree get a pass-2 targeted re-read; agreement feeds the calibrated confidence.
4. Review: policy decides everything above its per-field thresholds; the rest waits for a person. Decisions append; corrections write decided_value, scoped aliases, and calibration data.
5. Commit: one transaction inserts the materializations row, the obligation (or a revision through the materiality gate), lines with their order-line matches, evidence bindings, and outbox events. Derive projects FK columns from primary evidence bindings.
6. ERP: an erp_commands row per integration, executed idempotently against the customer's books.

Merged PDF, five documents inside

1. Intake creates one delivery, one artifact, and a file-level run, opened before its documents are split (the CHECK on runs allows exactly this case). No document row yet: nothing is known to be a document.
2. Classification proposes each document's type and page range, one proposal per segment with its confidence. Policy decides them; a low-confidence segmentation pauses for review before anything is sliced.
3. Materialize: per decided segment, create a derived file for each approved page range into renditions, insert a documents row with artifact_part and the parent delivery's provenance (the record of where a file or value came from, when it arrived, and who or what produced it), and start a per-document run. A redelivered split re-runs against the same digests and no-ops.
4. Each document proceeds through the happy path independently. A reclassification of the artifact replaces the earlier set while retaining it in history.

Workbooks follow the same flow with sheet names as artifact_part. There is no parent pseudo-document and no fake system actor: The delivery record carries the source information.

Denied segmentation: retry and correction

What happens when the split proposal is wrong. The cost depends entirely on timing: before documents exist, denial is free; after, it is a generation supersession. The gate exists to make the second case rare.

flowchart TD
    P["segmentation proposal<br/>on the file-level run (pass 1)"] --> D{"decision"}
    D -- "reject: wrong, machine<br/>should try again" --> E["escalated re-read:<br/>higher DPI, stronger model"]
    E --> P2["new proposal, pass+1,<br/>same run - prior pass kept"]
    P2 --> D
    D -- "defer" --> W["file run stays<br/>waiting_review"]
    W --> D
    D -- "auto-accept<br/>(policy threshold met)" --> M["materialize: documents born typed<br/>+ per-document runs"]
    D -- "correct: the reviewer's boundaries<br/>become the decided_value" --> M
    M --> OK(["pipeline continues<br/>per document"])
    OK -.-> L["split found wrong LATER,<br/>documents already exist"]
    L --> R["explicit resegment:<br/>new file-level run"]
    R --> NP["new segmentation proposal"]
    NP --> ND{"decision"}
    ND -- "accept / correct" --> G["generation+1 documents,<br/>born typed from the corrected split"]
    G --> SUP["old generation superseded, not deleted:<br/>runs, proposals, decisions, bindings kept;<br/>materialized records reconciled by supersession"]

Reject cannot loop unbounded: each escalated pass produces a new proposal that costs a fresh decision, and correct is the terminal fallback - the reviewer states the truth and documents are created from it. Every wrong guess stays on record as a labeled example.

Duplicate arrival, and re-upload with intent

If the resent file has the same hash, reuse the existing file record: a new deliveries row records the arrival (sender, channel, time) and nothing reprocesses - existing documents keep their original producing delivery and are not retargeted. A manual re-upload whose request metadata states what the operator wants also reuses the stored file: a reprocess request opens new runs on the existing documents; only an explicit reclassify or resegment creates a replacement document generation. Content is stored once in both cases; the difference between "we received it again" and "the operator wants it re-read" is a field on the delivery row rather than a code path.

Revised bill

The new document lands in the same document_family by chain key (the fields used to decide that two documents are versions of the same bill). The materiality gate compares with the current version: an empty or trivial diff returns the head unchanged, so a re-upload cannot reset an approved bill. A material diff creates a revision proposal; the decision (policy or human, per the tenant's versioning policy) inserts an obligation_revisions row, demotes the old head in the same transaction, and payments stay bound to the obligation. An older document is stored as a non-current revision without changing the current version or its payments.

BOL creates a shipment

1. Extraction and resolution produce proposals for the BL number, parties, ports, containers, and an entity_create proposal for the shipment itself, since no matching shipment exists.
2. On acceptance, commit create,s the shipment, its legs and handling units (trackable containers, packages, or pallets) in one transaction, with the materializations digest guarding redelivery. A container number that fails its check digit (a validation digit used to detect mistyped container numbers) materializes the unit with a null identifier and a pending field task; keep the suspected number only as an unapproved proposal; do not use it for tracking.
3. An evidence binding of kind source records that this document created this shipment. It does not populate any derived shipment field; a duplicate BOL arriving later gets an evidence binding only.
4. Tracking initialization appends the first observations; the reducer computes initial state; from here provider polling (regularly requesting tracking updates) and the BOL disagreeing about dates is the reducer's problem, under its versioned precedence rules (versioned rules deciding which source wins when dates conflict).

Ambiguous shipment link

Two candidate shipments for one certificate: The proposed match stores both candidate shipments and the evidence used to compare them including the document's own dates, so a container reused eight months apart resolves without a person. A genuine tie leaves the link proposal pending; the human review task shows both candidates with evidence; the decision creates the evidence binding, and derive updates the projections. A later manual re-link supersedes that specific binding and re-derives. No scheduled retry calls and charges the model again while review is pending while the task sits open.

A correction that teaches

1. An operator corrects the seller on a pending entity-match proposal: the decision row records decided_value beside the untouched proposal, the actor, and the task.
2. The same transaction writes a scoped alias ("this counterparty's 'ACME TRDG LLC.' means party X") with created_from_decision_id, subject to the an alias may identify only one record; a conflicting alias opens a review task; a collision with an existing alias opens an ambiguity task instead of silently overwriting.
3. The proposal-decision pair becomes a labeled example: it joins the calibration set (examples used to measure confidence accuracy) for that field and counterparty, and the few-shot retrieval (selecting relevant worked examples for each model prompt) pool (examples eligible for inclusion in later prompts) for the next document from that sender, with evaluation cases kept separate.
4. The next invoice from this supplier matches the seller exactly from the saved alias before calling a model, before any model runs.

Reprocess and change type

Reprocess opens a new run; the old run and its proposals stay, marked superseded. New proposals are decided (prior human decisions auto-apply where subject and value match; the rest re-enter review), commit re-materializes through the same digests, updating rows in place where the target exists, and evidence bindings are superseded and recreated rather than deleted. A change of type does not update document_type in place: it is a human-decided classification proposal that creates a replacement document generation, supersedes the old one, and reconciles its evidence bindings and materialized records. Nothing is hard-deleted, so a reprocess cannot destroy corrections, and the a safeguard requiring approval when changing the type would leave a shipment without its creating document is a policy rule: a type change that would orphan a shipment routes to an approval task instead of being forbidden in code.

ERP two-sided conflict

1. The inbound pull detects that both Aradus and the ERP changed the same record since the last sync. The erp_links row is marked as a conflict; the conflict remains paused until a person resolves it; routine synchronization must not resume it, since the park's own bookkeeping must not un-park it on the next sweep.
2. An adjudication task of kind erp_conflict opens with both versions and the field diff. No writes flow in either direction while it is open.
3. The operator picks theirs, ours, or a merge. The decision releases the conflict, and the outcome becomes an erp_commands row (ours) or an inbound patch (theirs), each idempotent under the customer's compatibility contract (customer-specific rules that preserve the ERP behavior their accounting processes rely on).

5 · Deep dives

Confidence and calibration

The gate does not read a model's self-assessment. Inputs to the calibrated score: cross-pass agreement (pass 2 uses a different model or rendering, since two correlated passes are weak evidence), deterministic validator results (results from fixed validation rules), resolver features (saved signals used when matching records), and the counterparty's track record. The calibrator maps raw scores to observed correction rates (adjusts model scores using how often similarly scored values were actually corrected) per document type, field and layout. Its training data has three sources by design: gold cases (manually verified examples with known correct answers), production corrections, and random audits of auto-accepted work, because a dataset containing only corrections is biased toward cases that were already sent to reviewers. Hierarchical priors cover cold starts (when a new customer or layout has little data, begin with broader field- and document-type statistics, then adapt as local evidence accumulates); retrieval examples are kept out of evaluation sets (examples shown to the model must not also be used to measure its accuracy).

Policy versus invariant

Configuration is applied from platform defaults down to field-specific rules (platform → tenant → counterparty and role → document type → layout → channel → field). It holds statements of business fact: templates with field-to-column maps, checks with tolerances and severities, matching strategy order, linkage conventions, materiality rules, autonomy grants. Use centrally managed thresholds based on observed error rates; customer-specific thresholds would overfit small datasets. Incident-derived correctness rules are not policy at all: the ordering ladder, tracking precedence and ERP idempotency are versioned code with fixtures (rules implemented in versioned code and protected by fixed regression examples), and policy can only choose whether an outcome routes to review.

Idempotency where money moves

Three ledgers make repeated delivery harmless (three durable operation records prevent retries from repeating billable or financial effects). model_invocations: a call is reserved under a deterministic ID before dispatch (record the call under a repeatable ID before sending it; after a crash, confirm the provider's result instead of charging again; billing uses completed calls rather than message deliveries), so a crash after the provider answered is reconciled, not re-billed, and billing reads this table rather than queue traffic. materializations: A repeated commit matches the existing input hash and makes no change. erp_commands: every push is a durable command with attempts (a stored ERP request with a retry history), and the customer-facing semantics (external ID schemes, for products first created in the ERP, treat an exact SKU match as the same product, case-insensitive number matching within tenant and direction, draft-only posting, already-posted no-ops, line replacement, conflicts that stay paused until a person resolves them) are a per-customer compatibility contract, changed only after certification against that customer's system.

Why Python survives, and shrunk to what

Model calls are just HTTP with a JSON schema; Go could make them. Python stays for two narrow reasons: PDF and image tooling (rasterization, DPI control, region crops for pass-2 re-reads (rendering PDF pages as images at a controlled resolution and rereading selected page regions)) where the Python ecosystem is years ahead, and the model-iteration loop, since few-shot assembly, eval tooling and DSPy-class prompt optimization are Python-first (python has stronger tools for selecting prompt examples, evaluating results, and automatically improving prompts) and that loop is where the accuracy advantage accumulates. The discipline that makes the split cheap: the gateway is stateless, has no database credentials, and every schema is defined once in JSON Schema and generated into both runtimes. The current system's Pydantic-plus-Go-struct duplication, which is why templates cannot be authored today, is the architecture prevents this duplication.

6 · Build plan at 100% feature parity

Parity is defined in user terms before anything is built: A machine-testable list of every user-visible capability (channels, duplicate rules, file types, document types classified as domain-producing or evidence-only, fields, resolutions, mutations, links, reviews, statuses, notifications, metering, tracking rules, ERP operations, retry and give-up paths, permissions, audit). Incident rules become characterization tests (tests that capture the current behavior - including incident cases - so the replacement cannot regress it) carrying their original incident examples. Table shapes, the the two current, inconsistent ways of identifying versions of the same bill and creating business records without committing all related changes atomically are details of the current implementation, not required user behavior, not parity.

First build one minimal end-to-end workflow and its shared foundations; then add complete document types one at a time. The spine is the honest name for the first deliverable, because that is where schedule risk lives: deliveries, artifacts, decomposition, runs, proposals, decisions, policy, the invocation ledger (the durable record of every model call and its cost), a review UI that shows the source page and the highlighted value, commit, outbox, audit, supporting one intentionally minimal end-to-end document flow. Then deliver each document type end to end, including its review UI, since semantic errors show up in review UX, not backend tests: supplier bills first (revisions, materiality, obligations, ERP posting), then BOL (shipment creation, tracking reduction) and customer PO (buyer layouts, allocations), then remaining domain-producing families, then the the remaining document types that provide supporting evidence but create no business records as configuration. The eval corpus and CI gates (automated checks that must pass before a change can merge or deploy) exist from the spine onward; each slice adds its gold cases before its model calls ship.

Migration uses the the allowed maintenance-window duration honestly. One global maintenance window (per-tenant windows on a shared database are would require the old and new systems to operate simultaneously): pause intake, tracking, workflows and outbound connectors (integrations that send updates to ERPs, tracking providers, or notification systems); snapshot; run migration scripts previously tested against production-like snapshots; validate; start the new system with keep outbound ERP updates paused; release connectors customer by customer after read-only verification against their live system. History that cannot honestly carry new-model provenance (extraction JSON without coordinates or model versions, overwritten tracking rows) imports as tagged legacy baselines and synthetic observations (import old values as explicitly marked starting data, and represent overwritten tracking history as generated records rather than presenting them as original source evidence) rather than invented provenance. Every non-deterministic case lands in an exception ledger (a queue of migration cases requiring human review) for human adjudication. Legacy IDs persist in a mapping registry (a table mapping legacy IDs to new-system IDs). Rollback is defined, not assumed: either writes stay blocked until the commitment point (the moment rollback stops being simple; any change after the switchover is recorded so recovery can restore to a defined point), or every post-cutover mutation is journaled with a stated recovery point.

Produced 4 Sep 2026 against branch next. Round-1 answers, cross-reviews and merged drafts are preserved in the session working files. Companion pages: Document Pipeline ERD (entity-relationship diagram) (the brownfield V0/V1 path (the migration path from the existing system through versions V0 and V1)) and Pipeline Write Paths.