Skip to content

Quote-Matching Pipeline — Step-by-Step Tutorial

A guided, chapter-by-chapter walkthrough that takes you from an empty checkout to a working system: your externally-run Postgres (catalog graph), Ollama (local models), and n8n (orchestration), plus the deterministic Python core that does the matching.

By the end you will understand every moving part, have the pieces connected, and know exactly what to configure to point it at your own catalog and mailbox.

Infrastructure runs separately. Postgres, Ollama, and n8n are not managed by this repo's Docker Compose — you run them your own way (existing instances, separate compose files, managed services, etc.). This project connects to them via configuration and provides a helper to initialize the database. The only optional container here is a small tools image for running the Python core.

What this tutorial covers: connecting to your infrastructure, initializing the schema, and the deterministic core. The two LLM steps (email → facts, facts → draft prose) and the hybrid-search service are wired on top — the tutorial shows exactly where they plug in, and flags the pieces that depend on your own data/infra decisions.

Table of contents

Chapter 0 — The big picture

The system turns a customer email ("I need a fire-rated partition for an office stairwell, ~24 m², 2.8 m tall") into a draft reply a salesperson reviews: the right system, its bill of materials, and any clarifying questions.

Two decisions shape everything (full rationale in architecture.md):

  1. Vector search finds the entry point; the graph delivers the relationships. Embeddings pick which system the email is about; the exact bill of materials is read from SQL relation tables, never guessed by a model.
  2. Thin model, thick scaffold. The LLM is used only for the two genuinely hard/quality-sensitive steps. Everything else — requirements, matching, BOM math, validation — is deterministic, auditable code.
Gmail trigger → preprocess → EXTRACT(LLM) → rules → match → validate → DRAFT(LLM) → Gmail draft + log
                                    │           └──────── deterministic core ────────┘
                                    └─ observable facts only (constrained JSON)

The three infrastructure services — which you run separately — and their usual ports:

Service Role Usual port
Postgres catalog graph + card embeddings (pgvector) 5432
Ollama local LLM + embedding backend (dev) 11434
n8n orchestration (the arrow diagram above) 5678

This repo does not start these; it connects to them. If you don't already have them, spin them up however you prefer (each ships an official image/binary).

Chapter 1 — Prerequisites

You provide the infrastructure (running separately):

  • Postgres 16+, ideally with the pgvector extension available (for semantic search; lexical search works without it).
  • Ollama (dev LLM + embeddings), or a frontier API for the LLM steps.
  • n8n (orchestration).
  • ~8 GB free RAM on the Ollama host for the dev models (a 7B quantized model fits an 8 GB Mac).

On your workstation:

  • Git.
  • Python 3.11+ to run the deterministic core, or Docker if you prefer the optional tools container.
  • psql on PATH to initialize the database (or use the tools container).

Check:

psql --version
python --version   # or: docker compose version

Chapter 2 — Get the code and tour the layout

git clone https://github.com/aaldescu/quote-match.git
cd quote-match

The parts you will touch:

docker-compose.yml       deploy the docs site (Dokploy + Traefik: domain + TLS)
docker-compose.local.yml local docs site (host port) + tools container
.env.example             configuration template → copy to .env
db/
  apply.sh             initialize an external Postgres (schema + pgvector + seed)
  migrations/          Postgres schema (the catalog graph)
  docker/              pgvector enablement (optional step)
  seed/                illustrative sample catalog
src/quote_match/        the deterministic core (rules, matching, validation, cards)
examples/demo.py        runnable end-to-end demo (no infra needed)
docs/                   architecture, pipeline wiring, this tutorial

If you just want to see it work with zero setup, jump to Chapter 7 — the core runs without any services. Otherwise continue in order.

Chapter 3 — Configuration (.env)

Copy the template and edit it:

cp .env.example .env

Point it at your running services:

Variable What it is Do this
DATABASE_URL connection to your external Postgres set host/port/user/pass/db
OLLAMA_URL your Ollama endpoint e.g. http://localhost:11434
EMBEDDING_MODEL Ollama embedding model bge-m3 (multilingual default)
EXTRACTION_MODEL model for email → facts (dev) qwen2.5:7b-instruct
DRAFT_MODEL model for the customer draft (dev) qwen2.5:7b-instruct
ANTHROPIC_API_KEY optional frontier key for hard steps leave blank to run fully local
APP_DATABASE_URL / APP_OLLAMA_URL same, as seen from inside the tools container only if you use the app container

DATABASE_URL / OLLAMA_URL are for commands you run on your host (migrations, the core). The APP_* variants exist because inside a container localhost means the container itself, so they default to host.docker.internal — change them if your services live on another host.

Chapter 4 — Connect to your infrastructure

You run Postgres, Ollama, and n8n separately. This chapter just confirms the tooling can reach them; make sure all three are up first.

Create the database (once) on your Postgres server, if it doesn't exist:

createdb -h <pg-host> -U <pg-user> quotematch
# or: psql -h <pg-host> -U <pg-user> -c 'CREATE DATABASE quotematch;'

Check connectivity using the values from your .env:

set -a; source .env; set +a          # load DATABASE_URL, OLLAMA_URL, ...

psql "$DATABASE_URL" -c "SELECT version();"        # Postgres reachable?
curl -s "$OLLAMA_URL/api/tags" | head -c 200        # Ollama reachable?

Open your n8n instance in a browser (commonly http://localhost:5678) and make sure you can log in. Note the hostnames n8n will use to reach Postgres and Ollama on your network — you'll need them in Chapter 9.

Chapter 5 — Initialize the database

Because Postgres runs separately, initialization is a one-time manual step. The helper applies everything in the right order:

set -a; source .env; set +a
bash db/apply.sh                 # schema + pgvector + sample seed

Options:

  • bash db/apply.sh --no-vector — skip pgvector (lexical search still works).
  • bash db/apply.sh --no-seed — schema only, no sample data.
  • No psql locally? Run it through the tools container: docker compose -f docker-compose.local.yml run --rm app bash db/apply.sh.

What it applies:

  1. db/migrations/0001_init.sql — the catalog graph (products, systems, procedures, edges).
  2. db/migrations/0002_cards_and_search.sql — system cards + lexical search.
  3. db/docker/03_pgvector.sql — enables pgvector and adds the embedding column (skipped with --no-vector).
  4. db/seed/sample_catalog.sql — the illustrative W111/W112 data (skipped with --no-seed).

Verify it loaded:

psql "$DATABASE_URL" -c \
  "SELECT code, attributes->>'fire_rating' AS fire FROM system ORDER BY code;"

Expected:

  code   | fire
---------+------
 W111    |
 W112    | EI60
 W112-MR | EI60

pgvector not installed on your server? CREATE EXTENSION vector will fail. Either install pgvector, or run with --no-vector and use lexical search only until it's available.

Re-applying: the scripts are additive. To start clean, drop and recreate the database (or TRUNCATE the tables) before re-running.

Chapter 6 — Pull the models into your Ollama

On your Ollama host, pull the dev models (a few GB the first time):

ollama pull bge-m3               # embeddings (multilingual)
ollama pull qwen2.5:7b-instruct  # extraction + draft

(If Ollama runs in its own container, docker exec <ollama> ollama pull ....)

List what you have and smoke-test embeddings against your endpoint:

curl -s "$OLLAMA_URL/api/tags" | grep -o '"name":"[^"]*"'

curl -s "$OLLAMA_URL/api/embeddings" \
  -d '{"model":"bge-m3","prompt":"fire-rated office partition"}' | head -c 120

You should get a JSON array of floats back. That confirms the embedding backend the hybrid-search layer will use (Chapter 8).

Why these models: bge-m3 is multilingual (matches mixed-language inboxes) and CPU-friendly. qwen2.5:7b-instruct supports structured/JSON output for constrained extraction and is small enough for an 8 GB dev box. Swap either via .env. In production the hard extraction and the customer draft can move to a frontier API — see Chapter 10.

Chapter 7 — Run the deterministic core

This is the heart of the system, and it needs no database and no models.

With Docker (using the tools profile):

docker compose -f docker-compose.local.yml run --rm app pytest          # 24 tests
docker compose -f docker-compose.local.yml run --rm app python examples/demo.py

Or directly on your machine:

pip install -e ".[dev]"
pytest
python examples/demo.py

The demo feeds three hand-written "extracted facts" through the whole non-LLM path and prints what each stage decides. For example, a bathroom request:

Derived requirements:  {'moisture_resistant': True}
    - Wet room -> impregnated (moisture-resistant) board. (wet_room_moisture)
Chosen: W112-MR
Draft payload:
  BOM:
    -    52.8 m2    Moisture-resistant plasterboard 12.5 mm (GB-MR-12.5)
    -   225.0 piece Drywall screw TN 25 mm (SC-TN-25)
    ...

Read examples/demo.py alongside the output — it is the clearest map of rules → match → validate → draft. The same logic is one call: quote_match.pipeline.run_deterministic(facts, candidates).

Chapter 8 — Generate and embed system cards (hybrid search)

Retrieval is two moves: hybrid search narrows the whole catalog to the top 5–10 candidate systems, then the graph expands those into exact BOMs.

The card is what search operates on. Generate one for any system:

docker compose -f docker-compose.local.yml run --rm app python -c \
 "from quote_match.catalog.cards import render_card; \
  from quote_match.catalog.sample_data import W112; print(render_card(W112))"

To make cards searchable you populate the system_card table:

  • Lexical half (works today): insert the card body; the body_tsv generated column + GIN index give you BM25-style ranking via ts_rank with no extra service.
  • Semantic half: embed the card body with bge-m3 (Chapter 6) and store the vector in system_card.embedding (the pgvector column added in Chapter 5). Query with embedding <=> $query_vector (cosine).

A minimal indexing loop (pseudocode for the service you'll add):

for system in catalog:
    body = render_card(system)
    if content_hash(body) unchanged: continue
    vec = ollama.embed(EMBEDDING_MODEL, body)
    upsert system_card(system_id, body, embedding=vec, content_hash=...)

Status: the schema, card generator, and lexical index ship now. The thin embedding+search service (a loop like above plus a search_catalog(query) endpoint) is the next component to build; its exact shape depends on the open questions in Appendix B, so it is intentionally left for you to wire once those are answered.

Chapter 9 — Wire the n8n workflow

Open your n8n instance and build the pipeline as nodes. Use whatever hostnames reach Postgres and Ollama on your network (the ones you confirmed in Chapter 4) — e.g. http://<ollama-host>:11434 and <pg-host>:5432. If n8n and the other services share a Docker network, use their service names; if n8n is on the host and they're elsewhere, use those addresses.

The flow (each arrow = one or a few nodes):

  1. Gmail/IMAP trigger — filter to your quotes label/alias.
  2. Preprocess (Code node) — strip signatures, quoted history, HTML → clean fresh-message text.
  3. Extract (HTTP → Ollama, or a frontier API) — call the model with the constrained JSON schema. Get the schema: 
    docker compose -f docker-compose.local.yml run --rm app python -c \
 "from quote_match.extraction.prompts import guided_json_schema; \
  import json; print(json.dumps(guided_json_schema()))"
    Pass it as Ollama's format (structured output) so only legal enum values come back. System/user prompts live in quote_match.extraction.prompts.
  4. Deterministic core — run rules → match → validate → draft assembly. Wrap pipeline.run_deterministic in a small HTTP service and call it from an n8n HTTP node, or run it via the tools container. This is the boxed core from Chapter 0; it needs the DATABASE_URL to your external Postgres.
  5. Draft (HTTP → model) — feed the assembled DraftContext to the draft prompt (quote_match.draft.draft_prompt). The prompt forbids inventing products, quantities, or standards.
  6. Gmail: create draft in-thread — never auto-send. The salesperson edits and sends.
  7. Log (Postgres/Sheets node) — sender, matched system, confidence, thread link.

Configure Gmail/IMAP and any API keys as n8n credentials (n8n → Credentials), not in this repo's .env. Export your finished workflow to n8n/workflow.json so it's version-controlled.

Chapter 10 — Model placement and privacy

Where each task runs (decided in planning):

Task Difficulty Dev (this stack) Production
Email → observable facts hard Ollama Qwen 7B frontier API / Qwen 14B LoRA
Rules derivation trivial code code
Graph search + BOM easy SQL/code SQL/code
Candidate ranking easy code code / Qwen 14B–32B
Customer draft quality-sensitive Ollama Qwen 7B frontier API
Embeddings trivial bge-m3 (Ollama) bge-m3 (self-host)

To move a step to a frontier API, point that n8n node at the hosted endpoint and set ANTHROPIC_API_KEY (or your provider's key) in the node credentials.

Privacy: emails contain PII and flow to whatever runs extraction/draft. The catalog stays local regardless. If you use a hosted model but must keep PII local, strip PII before the API call and re-insert it locally when rendering the draft — the deterministic core only ever sees ExtractedFacts, never raw PII.

Chapter 11 — Load your own catalog

The sample data is illustrative placeholder data. To use your own:

  1. Model it as products (SKUs), systems (solutions), procedures, and the edges between them (BOM with quantity_per_unit, requires, compatible). The schema (db/migrations/0001_init.sql) is the contract.
  2. Write an importer from your source format (ERP export, manufacturer data, flat SKU list) into those tables. This is a thin adapter — the schema is ready; only the parsing differs by source. (Source format is open question #1 — see Appendix B.)
  3. Tune the rules table (src/quote_match/rules/rules.yaml) with a domain expert. The placeholder fire/acoustic values must be replaced with the national standards you actually adhere to.
  4. Regenerate and re-embed cards (Chapter 8) after loading.

Extraction enums (src/quote_match/extraction/schema.py) should be widened to your catalog's real vocabulary — every enum value should be something the matcher can act on.

Chapter 12 — Build the eval set and tune

Before tuning any model, measure. Assemble 30–50 annotated real emails (the email text + the correct ExtractedFacts). Run extraction over them and score field-by-field accuracy. This tells you whether the dev 7B model is good enough or whether a step needs a frontier API.

Every salesperson correction to a draft is a free label. Once a few hundred accumulate, LoRA-fine-tune the extraction model (e.g. Qwen 14B) on them.

Chapter 13 — Going to production

  • Serving: replace dev Ollama with vLLM on a rented GPU (Hetzner GX / RunPod) for throughput and structured-output (guided_json) support, then point OLLAMA_URL / the n8n model nodes at it.
  • Postgres: use a managed/backed-up instance; keep pgvector. Revisit Postgres-vs-Neo4j only if relationship traversal gets deep (open question #4).
  • n8n: run the self-hosted or cloud instance your ops prefers (open question #3); secure it behind auth and TLS.
  • Secrets: never commit .env; use your platform's secret store. Rotate the DB password and any API keys.
  • Safety: the draft is always a Gmail draft — keep the human in the loop.

Chapter 14 — Troubleshooting

Symptom Cause / fix
psql: could not connect Wrong host/port/creds in DATABASE_URL, or Postgres not reachable from your machine. Confirm with psql "$DATABASE_URL" -c 'select 1'.
db/apply.sh: type "..." already exists Schema already applied. Drop/recreate the DB (or TRUNCATE) to re-apply cleanly.
CREATE EXTENSION vector fails pgvector isn't installed on your server. Install it, or run bash db/apply.sh --no-vector.
Ollama model not found Pull it on the Ollama host: ollama pull <model> (Chapter 6).
Ollama endpoint refused Check OLLAMA_URL; if Ollama binds only to localhost, set OLLAMA_HOST=0.0.0.0 on its host so other machines/containers can reach it.
n8n can't reach Ollama/Postgres Use hostnames valid on n8n's network, not localhost (which is n8n itself). Same-network containers → service names; otherwise the real host/IP.
app container can't reach services Inside a container localhost is the container. Set APP_DATABASE_URL / APP_OLLAMA_URL (default host.docker.internal).
docker compose -f docker-compose.local.yml run app fails to build First build compiles deps; check docker compose -f docker-compose.local.yml build app. Needs network for the base image.
Tests pass but demo differs from real Sample data is illustrative; results reflect sample_data.py, not a real catalog.

Useful commands:

set -a; source .env; set +a
psql "$DATABASE_URL"                                  # inspect the catalog DB
curl -s "$OLLAMA_URL/api/tags"                        # what models are loaded
docker compose -f docker-compose.local.yml run --rm app pytest                    # run the core in-container

Appendix A — Configuration reference

All configuration lives in .env (copied from .env.example). Postgres, Ollama, and n8n run separately, so these are connection settings, not service definitions.

Variable Default Used by Meaning
DATABASE_URL postgresql://quote:change-me@localhost:5432/quotematch db/apply.sh, host scripts connection to your external Postgres
OLLAMA_URL http://localhost:11434 host scripts, indexing your Ollama endpoint
EMBEDDING_MODEL bge-m3 indexing/search embedding model
EXTRACTION_MODEL qwen2.5:7b-instruct extract node email → facts model
DRAFT_MODEL qwen2.5:7b-instruct draft node draft prose model
ANTHROPIC_API_KEY — optional frontier steps hosted-model key
APP_DATABASE_URL ...@host.docker.internal:5432/... app container Postgres as seen from inside the tools container
APP_OLLAMA_URL http://host.docker.internal:11434 app container Ollama as seen from inside the tools container

n8n's own configuration (its database, model endpoints, Gmail/IMAP credentials) is set inside n8n, since you run it separately.

Appendix B — Open questions

These were carried over from planning and gate specific pieces of the build. Answer them before the corresponding step:

  1. Catalog source format (flat SKU list? ERP export? manufacturer data?) → determines the importer (Chapter 11).
  2. Languages of incoming emails → confirms the embedding model (bge-m3 is the multilingual default) and the extraction prompt language(s).
  3. Where n8n runs (self-hosted vs cloud) → deployment (Chapter 13).
  4. Postgres vs Neo4j → defaulted to Postgres; revisit only if traversal depth demands it. The node/edge model ports either way.

See pipeline.md for how these map onto the remaining components.

Appendix C — Serve these docs (deploy to a VPS)

This tutorial can be published as a browsable website (MkDocs Material built into an nginx container). There are two compose files:

File For How it's exposed
docker-compose.yml Dokploy + Traefik on a VPS by domain, automatic TLS, no host port
docker-compose.local.yml local browsing / running the core host port DOCS_PORT (default 8090)

Locally

docker compose -f docker-compose.local.yml up -d --build docs
# browse http://localhost:8090   (change with DOCS_PORT in .env)

On a VPS with Dokploy (domain + automatic HTTPS)

The default docker-compose.yml is written for Dokploy's Traefik: it routes by domain over the shared dokploy-network and gets a Let's Encrypt certificate — no published host port, so it never collides with other apps.

  1. DNS: create an A record for your subdomain (e.g. quote-match.previewrun.de) pointing at the VPS IP.
  2. Dokploy app: create a Compose app from this repo (it uses ./docker-compose.yml).
  3. Environment: set DOMAIN_URL to that subdomain — it's interpolated into the Traefik Host(...) rules: 
    DOMAIN_URL=quote-match.previewrun.de
  4. Deploy. Traefik picks up the labels, requests the certificate, and serves the site at https://<DOMAIN_URL>.

To use a different subdomain, just change DOMAIN_URL (and its DNS record) — nothing in the compose file needs editing.

Updating content

Edit the Markdown under docs/ and redeploy (Dokploy) or rebuild locally:

docker compose -f docker-compose.local.yml up -d --build docs

The image is a static build served by nginx — small, with no runtime dependency on Python, the database, or the models. It's just the docs.

Editing live while writing: for a fast edit loop run the dev server instead of the container: pip install mkdocs-material && mkdocs serve → http://localhost:8000 with live reload.