Pipeline wiring (n8n) and next steps

The flow

Gmail/IMAP trigger (quotes label)
      │
      ▼
Preprocess            strip signatures, quoted history, HTML → clean fresh text
      │
      ▼
Extract (LLM)         constrained JSON → ExtractedFacts   [extraction/schema.py]
      │                (frontier API now; fine-tuned Qwen 14B later)
      ▼
┌─────────────────────────────────────────────────────────────┐
│  DETERMINISTIC CORE  (pipeline.run_deterministic)            │
│    Rules      facts → requirements          [rules/]         │
│    Match      hybrid search + graph → rank  [catalog/]       │
│    Validate   consistency checks            [validation/]    │
│    Assemble   → DraftContext                [draft/]         │
└─────────────────────────────────────────────────────────────┘
      │
      ▼
Draft (LLM)           DraftContext → polite prose         [draft/builder.py]
      │                (quality-sensitive; frontier API)
      ▼
Gmail draft in-thread (NEVER auto-send) + log to Sheets/CRM

The boxed core is this repo, callable as one function (quote_match.pipeline.run_deterministic). In n8n it can be a single Code / Execute-Command node, or split across nodes that call the same package.

Model placement (decided in planning)

Task	Difficulty	Where
Email → observable facts	hard	Frontier API (or fine-tuned Qwen 14B later)
Rules derivation	trivial	plain code (this repo)
Graph search + BOM expansion	easy	SQL / code (this repo)
Candidate ranking	easy	plain code now; self-hosted Qwen 14B–32B optional
Customer-facing draft	quality-sensitive	Frontier API
Embeddings	trivial	self-hosted bge-m3 / nomic-embed-text (CPU, multilingual)

Privacy. Emails carry PII and flow to the API side; the catalog stays local. If needed, strip PII before the API call and re-insert locally when rendering the draft. The deterministic core never needs to see raw PII — it operates on ExtractedFacts.

The hybrid-search service (not yet built — needs infra answers)

The one substantial component still to build is the retrieval service that turns a query into candidate systems:

1. Generate/refresh system_card rows (catalog/cards.render_card), embed with bge-m3, store in the embedding column (pgvector).
2. Query = a natural-language summary of the extracted facts. Retrieve by 0.5·cosine + 0.5·ts_rank (or RRF) over system_card → top 5–10.
3. Hand those candidates to catalog.matching.score_systems, then to validation and draft assembly.

db/migrations/0002_cards_and_search.sql already provides the lexical (body_tsv + GIN) half on vanilla Postgres; the vector half is a guarded block to run where pgvector is installed. The service itself is deferred because its shape depends on the open questions below.

Open questions carried over from planning

These were flagged in the design spec and still gate specific decisions:

1. Source format of the existing catalog (flat SKU list? ERP export? manufacturer data?) → determines the loader that populates product / system / edges. The schema is ready; the importer is a thin adapter once the format is known.
2. Languages of incoming emails → picks the embedding model (bge-m3 is multilingual, hence the default) and the extraction prompt language(s).
3. Where n8n runs (self-hosted vs cloud) → affects how it reaches this package and the Postgres instance.
4. Postgres vs Neo4j → defaulted to Postgres; revisit only if traversal depth demands it. The domain model ports either way.

Suggested next steps

1. Confirm the four open questions above.
2. Write the catalog importer (source format → schema) and load real data.
3. Stand up the embedding + hybrid-search service (section above).
4. Build the eval set: 30–50 salesperson-annotated real emails; measure extraction accuracy before tuning anything.
5. Wire the n8n workflow around run_deterministic.
6. Once a few hundred salesperson corrections accumulate, LoRA-fine-tune the extraction model (corrections are free labels).