Back to architecture
live

Knowledge Ingestion

Files, URLs, and catalogs become searchable knowledge.

Uploads are sorted by type. Web pages are rendered and cleanly extracted, catalog spreadsheets are mapped into structured product tables, and everything else is chunked and embedded into the vector store, so the bot can retrieve it later.

Data flow

How the pieces connect and where data moves.

Components: Upload or URL (files, sheets, sites), Ingestion (detect + route), URL scraper (Jina + BeautifulSoup), Catalog import (structured rows), Chunk + embed, pgvector (RAG chunks), Catalog tables (products + SKUs). Connections: Upload or URL to Ingestion; Ingestion to URL scraper for web pages; Ingestion to Catalog import for catalog sheets; Ingestion to Chunk + embed for docs + text; URL scraper to Chunk + embed for extracted text; Chunk + embed to pgvector for embeddings; Catalog import to Catalog tables for rows.

Loading data-flow map

Behavior

How it behaves step by step over time.

Loading behavior diagram
An upload or URL is detected by type. Web pages are scraped with a renderer plus HTML extraction, catalog sheets are mapped and cleaned into product tables, and documents or text are chunked and embedded into pgvector for retrieval.